An introduction to using R in Power BI via the various touch points such as: R script data sources, R transformations, custom R visuals, and the community gallery of R visualizations
Modern ETL: Azure Data Factory, Data Lake, and SQL DatabaseEric Bragas
In this presentation, we take a look at the components of a modern ETL platform using the latest and greatest Azure technologies to leverage PaaS services for parallel data loading, distributed data processing, and SQL databases as a semantic layer. Originally presented for the Orange County SQL Saturday, April 2018.
Part 3 - Modern Data Warehouse with Azure SynapseNilesh Gule
Slide deck of the third part of building Modern Data Warehouse using Azure. This session covered Azure Synapse, formerly SQL Data Warehouse. We look at the Azure Synapse Architecture, external files, integration with Azuer Data Factory.
The recording of the session is available on YouTube
https://www.youtube.com/watch?v=LZlu6_rFzm8&WT.mc_id=DP-MVP-5003170
An introduction to using R in Power BI via the various touch points such as: R script data sources, R transformations, custom R visuals, and the community gallery of R visualizations
Modern ETL: Azure Data Factory, Data Lake, and SQL DatabaseEric Bragas
In this presentation, we take a look at the components of a modern ETL platform using the latest and greatest Azure technologies to leverage PaaS services for parallel data loading, distributed data processing, and SQL databases as a semantic layer. Originally presented for the Orange County SQL Saturday, April 2018.
Part 3 - Modern Data Warehouse with Azure SynapseNilesh Gule
Slide deck of the third part of building Modern Data Warehouse using Azure. This session covered Azure Synapse, formerly SQL Data Warehouse. We look at the Azure Synapse Architecture, external files, integration with Azuer Data Factory.
The recording of the session is available on YouTube
https://www.youtube.com/watch?v=LZlu6_rFzm8&WT.mc_id=DP-MVP-5003170
Analyzing StackExchange data with Azure Data LakeBizTalk360
Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different. Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings.
If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.
Slidedeck related to the talk presented at the Manila Data Day event March 2020. The demo covers Azure services like Data Lake Storage (Gen 2), Azure Data Factory, Azure Databricks, Azure Synapse, Key Vault and Active directory to build a modern data warehouse.
10 Things Learned Releasing Databricks Enterprise WideDatabricks
Implementing tools, let alone an entire Unified Data Platform, like Databricks, can be quite the undertaking. Implementing a tool which you have not yet learned all the ins and outs of can be even more frustrating. Have you ever wished that you could take some of that uncertainty away? Four years ago, Western Governors University (WGU) took on the task of rewriting all of our ETL pipelines in Scala/Python, as well as migrating our Enterprise Data Warehouse into Delta, all on the Databricks platform. Starting with 4 users and rapidly growing to over 120 users across 8 business units, our Databricks environment turned into an entire unified platform, being used by individuals of all skill levels, data requirements, and internal security requirements.
Through this process, our team has had the chance and opportunity to learn while making a lot of mistakes. Taking a look back at those mistakes, there are a lot of things we wish we had known before opening the platform to our enterprise.
We would like to share with you 10 things we wish we had known before WGU started operating in our Databricks environment. Covering topics surrounding user management from both an AWS and Databricks perspective, understanding and managing costs, creating custom pipelines for efficient code management, learning about new Apache Spark snippets that helped save us a fortune, and more. We would like to provide our recommendations on how one can overcome these pitfalls to help new, current and prospective users to make their environments easier, safer, and more reliable to work in.
Moving to the cloud; PaaS, IaaS or Managed InstanceThomas Sykes
In this session we'll look at the cloud choices available in Azure for SQL Server. Whether it's PaaS, IaaS or Managed Instance we'll look into the features provided, the major differences and the Pros and Cons of each solution and how to choose the best option available.
Non è necessario tirare in ballo l’IoT per immaginare quanto possa essere utile per fare query sui dati mentre questi fluiscono verso il database, e non solamente dopo. Si apre un mondo di possibilità per quanto riguarda alerting & monitoring in tempo reale, che è chiaramente la parte più immediata, ma è anche possibile pensare a cose come real-time dasboarding e soluzioni per aggiustare prezzi ed offerte di prodotti in tempo reale. In questa sessione vedremo come è possibile utilizzare Azure Stream Analytics ed il suo linguaggio SQL-Like per analizzare i dati in streaming, e quindi iniziare a prendere confidenza con questo nuovo approccio ormai sempre pià in voga e sempre più richesto, sia nel mondo dell’IoT che non.
Integration Monday - Analysing StackExchange data with Azure Data LakeTom Kerkhove
Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different.
Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings.
If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.
comparison of Excel add-ins and other solutions for implementing data mining or machine learning solutions on the Microsoft stack - includes coverage of XLMiner, Analysis Services Data Mining and PredixionSoftware
Using Redash for SQL Analytics on DatabricksDatabricks
This talk gives a brief overview with a demo performing SQL analytics with Redash and Databricks. We will introduce some of the new features coming as part of our integration with Databricks following the acquisition earlier this year, along with a demo of the other Redash features that enable a productive SQL experience on top of Delta Lake.
The DBA landscaping and tooling is changing rapidly and the DBA need to keep up with the new technologies related to SQL Azure, Big Data and NoSQL. SQL Server 2016 is brining many new features that will help you become that modern DBA. This sessions will demo the new features help you get ready for the change that is coming in the world of Data.
Geek Sync | Taking Your First Steps to the Cloud—Building a Hybrid ModelIDERA Software
You can watch the replay for this Geek Sync webcast in the IDERA Resource Center: https://www.idera.com/resourcecentral/webcasts/geeksync/first-steps-to-cloud-hybrid-model
There are different approaches to migrating to the cloud, but most organizations start out in a hybrid fashion, with some resources on-premises and some in the cloud. Whether it’s using the cloud as a disaster recovery data center, or building out a full new deployment, during this session with Joey D'Antoni you will learn about what you need from a SQL Server perspective. Joey will also discuss other infrastructure components such as storage, networking and authentication. You will learn about different deployment and migration options and what steps you should take to get started with your cloud journey.
Speaker: Joey D’Antoni is a senior consultant and SQL Server MVP with over a decade of experience working in both Fortune 500 and smaller firms. He is a principal architect for Denny Cherry and Associates and lives in Malvern, PA. He is a frequent speaker at major tech events, and blogger about all topics technology. He believes that no single platform is the answer to all technology problems. He holds a BS in Computer Information Systems from Louisiana Tech University and an MBA from North Carolina State University. Joey is the co-author of the Microsoft white paper "Using Power BI in a Hybrid Environment."
Geek Sync | Deployment and Management of Complex Azure EnvironmentsIDERA Software
You can watch the replay of this Geek Sync webinar in the IDERA Resource Center: http://ow.ly/pg7N50A4svf.
Today's data management professional is finding their landscape changing. They have multiple database platforms to manage, multi-OS environments and everyone wants it now.
Join IDERA and Kellyn Pot’Vin-Gorman as she discusses the power of auto deployment in Azure when faced with complex environments and tips to increase the knowledge you need at the speed of light. Kellyn will cover scripting basics, advanced Portal features, opportunities to lessen the learning curve and how multi-platform and tier doesn't have to mean multi-cloud.
Attendees can expect to learn how to build automation scripts efficiently, even if you have little scripting experience, and how to work with Azure automation deployments. This session will allow you to begin building a repository of multi-platform development scripts to use as needed.
About Kellyn: Kellyn Pot’Vin-Gorman is a member of the Oak Table Network and an IDERA ACE and Oracle ACE Director alumnus. She is the newest Technical Solution Professional in Power BI with AI in the EdTech group at Microsoft. Kellyn is known for her extensive work with multi-database platforms, DevOps, cloud migrations, virtualization, visualizations, scripting, environment optimization tuning, automation, and architecture design. She has spoken at numerous technical conferences for Oracle, Big Data, DevOps, Testing and SQL Server. Her blog, http://dbakevlar.com and social media activity under her handle, DBAKevlar is well respected for her insight and content.
Slides from my talk at Big Data Conference 2018 in Vilnius
Doing data science today is far more difficult than it will be in the next 5-10 years. Sharing, collaborating on data science workflows in painful, pushing models into production is challenging.
Let’s explore what Azure provides to ease Data Scientists’ pains. What tools and services can we choose based on a problem definition, skillset or infrastructure requirements?
In this talk, you will learn about Azure Machine Learning Studio, Azure Databricks, Data Science Virtual Machines and Cognitive Services, with all the perks and limitations.
Analyzing StackExchange data with Azure Data LakeBizTalk360
Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different. Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings.
If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.
Slidedeck related to the talk presented at the Manila Data Day event March 2020. The demo covers Azure services like Data Lake Storage (Gen 2), Azure Data Factory, Azure Databricks, Azure Synapse, Key Vault and Active directory to build a modern data warehouse.
10 Things Learned Releasing Databricks Enterprise WideDatabricks
Implementing tools, let alone an entire Unified Data Platform, like Databricks, can be quite the undertaking. Implementing a tool which you have not yet learned all the ins and outs of can be even more frustrating. Have you ever wished that you could take some of that uncertainty away? Four years ago, Western Governors University (WGU) took on the task of rewriting all of our ETL pipelines in Scala/Python, as well as migrating our Enterprise Data Warehouse into Delta, all on the Databricks platform. Starting with 4 users and rapidly growing to over 120 users across 8 business units, our Databricks environment turned into an entire unified platform, being used by individuals of all skill levels, data requirements, and internal security requirements.
Through this process, our team has had the chance and opportunity to learn while making a lot of mistakes. Taking a look back at those mistakes, there are a lot of things we wish we had known before opening the platform to our enterprise.
We would like to share with you 10 things we wish we had known before WGU started operating in our Databricks environment. Covering topics surrounding user management from both an AWS and Databricks perspective, understanding and managing costs, creating custom pipelines for efficient code management, learning about new Apache Spark snippets that helped save us a fortune, and more. We would like to provide our recommendations on how one can overcome these pitfalls to help new, current and prospective users to make their environments easier, safer, and more reliable to work in.
Moving to the cloud; PaaS, IaaS or Managed InstanceThomas Sykes
In this session we'll look at the cloud choices available in Azure for SQL Server. Whether it's PaaS, IaaS or Managed Instance we'll look into the features provided, the major differences and the Pros and Cons of each solution and how to choose the best option available.
Non è necessario tirare in ballo l’IoT per immaginare quanto possa essere utile per fare query sui dati mentre questi fluiscono verso il database, e non solamente dopo. Si apre un mondo di possibilità per quanto riguarda alerting & monitoring in tempo reale, che è chiaramente la parte più immediata, ma è anche possibile pensare a cose come real-time dasboarding e soluzioni per aggiustare prezzi ed offerte di prodotti in tempo reale. In questa sessione vedremo come è possibile utilizzare Azure Stream Analytics ed il suo linguaggio SQL-Like per analizzare i dati in streaming, e quindi iniziare a prendere confidenza con questo nuovo approccio ormai sempre pià in voga e sempre più richesto, sia nel mondo dell’IoT che non.
Integration Monday - Analysing StackExchange data with Azure Data LakeTom Kerkhove
Big data is the new big thing where storing the data is the easy part. Gaining insights in your pile of data is something different.
Based on a data dump of the well-known StackExchange websites, we will store & analyse 150+ GB of data with Azure Data Lake Store & Analytics to gain some insights about their users. After that we will use Power BI to give an at a glance overview of our learnings.
If you are a developer that is interested in big data, this is your time to shine! We will use our existing SQL & C# skills to analyse everything without having to worry about running clusters.
comparison of Excel add-ins and other solutions for implementing data mining or machine learning solutions on the Microsoft stack - includes coverage of XLMiner, Analysis Services Data Mining and PredixionSoftware
Using Redash for SQL Analytics on DatabricksDatabricks
This talk gives a brief overview with a demo performing SQL analytics with Redash and Databricks. We will introduce some of the new features coming as part of our integration with Databricks following the acquisition earlier this year, along with a demo of the other Redash features that enable a productive SQL experience on top of Delta Lake.
The DBA landscaping and tooling is changing rapidly and the DBA need to keep up with the new technologies related to SQL Azure, Big Data and NoSQL. SQL Server 2016 is brining many new features that will help you become that modern DBA. This sessions will demo the new features help you get ready for the change that is coming in the world of Data.
Geek Sync | Taking Your First Steps to the Cloud—Building a Hybrid ModelIDERA Software
You can watch the replay for this Geek Sync webcast in the IDERA Resource Center: https://www.idera.com/resourcecentral/webcasts/geeksync/first-steps-to-cloud-hybrid-model
There are different approaches to migrating to the cloud, but most organizations start out in a hybrid fashion, with some resources on-premises and some in the cloud. Whether it’s using the cloud as a disaster recovery data center, or building out a full new deployment, during this session with Joey D'Antoni you will learn about what you need from a SQL Server perspective. Joey will also discuss other infrastructure components such as storage, networking and authentication. You will learn about different deployment and migration options and what steps you should take to get started with your cloud journey.
Speaker: Joey D’Antoni is a senior consultant and SQL Server MVP with over a decade of experience working in both Fortune 500 and smaller firms. He is a principal architect for Denny Cherry and Associates and lives in Malvern, PA. He is a frequent speaker at major tech events, and blogger about all topics technology. He believes that no single platform is the answer to all technology problems. He holds a BS in Computer Information Systems from Louisiana Tech University and an MBA from North Carolina State University. Joey is the co-author of the Microsoft white paper "Using Power BI in a Hybrid Environment."
Geek Sync | Deployment and Management of Complex Azure EnvironmentsIDERA Software
You can watch the replay of this Geek Sync webinar in the IDERA Resource Center: http://ow.ly/pg7N50A4svf.
Today's data management professional is finding their landscape changing. They have multiple database platforms to manage, multi-OS environments and everyone wants it now.
Join IDERA and Kellyn Pot’Vin-Gorman as she discusses the power of auto deployment in Azure when faced with complex environments and tips to increase the knowledge you need at the speed of light. Kellyn will cover scripting basics, advanced Portal features, opportunities to lessen the learning curve and how multi-platform and tier doesn't have to mean multi-cloud.
Attendees can expect to learn how to build automation scripts efficiently, even if you have little scripting experience, and how to work with Azure automation deployments. This session will allow you to begin building a repository of multi-platform development scripts to use as needed.
About Kellyn: Kellyn Pot’Vin-Gorman is a member of the Oak Table Network and an IDERA ACE and Oracle ACE Director alumnus. She is the newest Technical Solution Professional in Power BI with AI in the EdTech group at Microsoft. Kellyn is known for her extensive work with multi-database platforms, DevOps, cloud migrations, virtualization, visualizations, scripting, environment optimization tuning, automation, and architecture design. She has spoken at numerous technical conferences for Oracle, Big Data, DevOps, Testing and SQL Server. Her blog, http://dbakevlar.com and social media activity under her handle, DBAKevlar is well respected for her insight and content.
Slides from my talk at Big Data Conference 2018 in Vilnius
Doing data science today is far more difficult than it will be in the next 5-10 years. Sharing, collaborating on data science workflows in painful, pushing models into production is challenging.
Let’s explore what Azure provides to ease Data Scientists’ pains. What tools and services can we choose based on a problem definition, skillset or infrastructure requirements?
In this talk, you will learn about Azure Machine Learning Studio, Azure Databricks, Data Science Virtual Machines and Cognitive Services, with all the perks and limitations.
Valkuilen en successen binnen een implementatie van Office 365, SharePoint, Exchange en Lync Online in de Cloud. 4000 seats binnen Zorg - Health. Intramurele en extramurele medewerkers hebben samen een digitale werkomgeving. Het Nieuwe Werken voor een thuiszorginstelling.
Building Modern Data Platform with Microsoft AzureDmitry Anoshin
This presentation will cover Cloud history and Microsoft Azure Data Analytics capabilities. Moreover, it has a real-world example of DW modernization. Finally, we will check the alternative solution on Azure using Snowflake and Matillion ETL.
Azure Synapse Analytics is Azure SQL Data Warehouse evolved: a limitless analytics service, that brings together enterprise data warehousing and Big Data analytics into a single service. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources, at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. This is a huge deck with lots of screenshots so you can see exactly how it works.
Power Platform Leeds - November 2019 - Microsoft Ignite AnnouncementsSimon Doy
Slide deck by Simon Hudson and Simon Doy.
Provides the key announcements from Microsoft Ignite 2019 on Power Platform, SharePoint, Microsoft Teams, One Drive, Azure and other things that we thought the user group would be interested in.
Machine learning applications are typically stitched together from hopes and dreams, shell scripts, cron jobs, home-grown schedulers, snippets of configuration clipped from multiple blog posts, thousands of hard-coded business rules, a.k.a. "our SQL corpus," and a few lines of training and testing code. Organizing all the moving parts into something maintainable and supportive of ongoing development is a challenge most teams have on their TODO list, roadmap, or tech debt pile. Getting ahead of the day-to-day demands and settling into a sane architecture often seems like an unattainable goal. The past several years have seen an explosion of tool-building in the data engineering and analytics area, including in Apache projects spanning the areas of search and information retrieval, job orchestration, file and stream formats, and machine learning libraries. In this talk we will cover our product and development teams' choices of architecture and tools, from data ingestion and storage, through transformations and processing, to presentation of results and publishing to web services, reports, and applications.
Office 365 development overview nilesh shah 24 mar 2018 webinarNilesh Shah
Office 365 development overview:
Why Office 365 development
Office 365 Developer Program
Ways to develop Office 365 Apps/Integration
Registering Office 365 Application
Tools for Office 365 development
Office 365 development overview Nilesh Shah 15 march 2018Nilesh Shah
Overview of Office 365 development
Office 365 Add-in development, coding and demo
Event: C# Corner Toronto chapter's monthly meetup organized at Microsoft Canada's Toronto downtown office.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Check out the webinar slides to learn more about how XfilesPro transforms Salesforce document management by leveraging its world-class applications. For more details, please connect with sales@xfilespro.com
If you want to watch the on-demand webinar, please click here: https://www.xfilespro.com/webinars/salesforce-document-management-2-0-smarter-faster-better/
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
We describe the deployment and use of Globus Compute for remote computation. This content is aimed at researchers who wish to compute on remote resources using a unified programming interface, as well as system administrators who will deploy and operate Globus Compute services on their research computing infrastructure.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...
Adf and ala design c sharp corner toronto chapter feb 2019 meetup nik shahriar
1. Presented by : Nik – Shahriar Nikkhah
Date : 2019-02-25 Toronto, Canada
Azure Data Factory And
Azure Logic Apps Design
2. • Microsoft Canada (MS-Mississauga)
Everyone at Microsoft
• C-Sharp Corner (Toronto)
Nilesh Shah (Toronto Chapter Lead)
Nik – Shahriar Nikkhah (Executive Member of Toronto Chapter)
Vivek Patel (Toronto Power BI Chapter Lead)
• SQL Data Side Inc.
Thank you…
And thank you to you all.
3. • Nik – Shahriar Nikkhah
Snr Data Engineer, Snr Azure Data Integration Lead/Design, Snr BI Consultant , Snr Technical
Team Lead
(IBM, LGS, SNC-Lavalin)
Microsoft Certified, MCSE, MCSA, MCP, MCITP, MCTS
Main focus: Big Data, Azure Data Factory, Azure Databrick, Azure Logic App, IoT Hub, Azure Stream Analytics,
Azure SQL Data Warehouse, Azure SQL Database, Azure Storage Account (Blob), Azure Data Lake Storage,
Azure Analysis Services, Power BI, SSIS, SQL Tabular DM, Azure PowerShell and On-Prem SQL Server
• SQLDataSide@gmail.com Tel: (647) 621-2242
• http://sqldataside.blogspot.ca/ https://www.linkedin.com/in/nnikkhah/
• https://www.c-sharpcorner.com/members/shahriar-nikkhah (Executive Member of Toronto Chapter)
About me
Please ask questions
4. • Nik - Shahriar Nikkhah
• Sr. Technical Team Lead
• Sr. BI Consultant
• C# Corner MVP
• Nilesh Shah
• Tech Lead, .NET & O365 dev.
• TOGAF 9.1 EA
• MS MVP, C# Corner MVP
• Vivek Patel
• Team Lead, Data & Analytics
• Microsoft Data Platform MVP
Toronto chapter – founder members
C# Corner
7. Agenda
We will be focusing on the above agenda
• What is Azure Data Factory (ADF)?
• What is Azure Logic Apps (ALA)?
• Unzip files in Logic Apps
• Looking from a different Angle.
• Typical Data Flow
• Unzip Files using ADF and ALA
• Mix and Match ADF ALA & ADB
• Unzip Demo
• Motivation slide
8. • Azure Subscription
• Azure Data Factory V2+
• Azure Storage Account (Blob Storage)
• Visual studio 2017+
• Azure tools for Visual Studio
• Microsoft Azure Storage Explorer
• Azure PowerShell
Requirements
https://azure.microsoft.com/en-ca/free/
9. • Cloud Data Integration Service
• Compose data storage, movement, …., processing service
into automated data pipelines
• ETL or ELT Tool in the cloud
• Azure Databricks Cluster
• Parallel processing
What is ADF?
Can we compare ADF with SSIS ?
10. • Cloud Data Integration Service
• Automate and orchestrate tasks, business process, Work
flows, When you need to integrate apps, data, systems,
and services across organisations.
• ETL or ELT Tool in the cloud
• No documentation found on Clustering for ALA
• Parallel processing
What is ALA?
Can we somehow ALA, ADF and SSIS?
11. Unzip files in Logic Apps
Do you see any limitation? and if so is ALA the right tool to unzip?
12. • Think as if you are working for google.
• Think big, think differently (Left join in google reports)
• Learn from scratch (How to create a table in ASDW in the
cloud vs creating a table in on-prem SQL Server)
• Profiling the data with different Tools (ADK/PBI).
• <= 2500 Pipelines, 200 ADF, 150 - 1000 ADB, 100
table/data dependency
• Replacing old features with new releases (Tool mature).
Looking From a Different Angle
How fast can you replace a old component with a new component(s)?
Considering the time & work needed (Dev, Test, QA, UAT, Deployment…)
13. Read files from a path
Filter out unwanted
files
Loop Files one by one
Typical Data Flow
Framework and prediction approach is critical in your design.
• 1 - GM_GetListOfFiles
• 2 - FL_SelectRequiredFilesOnly
• 3 - FE_LoopFiles
• Copy action is conducting in the loop
14. Unzip files using ADF and ALA
Two services with the same feature, what’s the difference/limitation?
15. Pipeline 2
Mix and Match ADF ALA & ADB
All parameters must have the same names.
Do they have the same parameters in the above samples?
Read files from a path
Filter out unwanted
files
Loop Files one by one
Pipeline 3
Pipeline 4
Pipeline 5
Pipeline 1
16. • Understand the Flow (Profiling Profiling Profiling).
• Covers all technical and non technical requirements (SOW).
• R&D, POC and get second opinion (by MS if possible).
• Rough sketch your Pipeline Flow (Categorise your ADFs).
• Plan a deployment plan with the devops team once a week.
• Stress test, run jobs everyday overnight, Check the logs, use
Azure Advisor, Azure Security Center, failure is the key.
• Predict massive changes & how easy you can change them.
• Reusability and Documentation (Cursor story)
Building ADF
The goal is to hand over the work to someone junior.
20. • IoT Hub DevKit from MS (temperature)
(Microsoft Canada HQ in Mississauga)
Webinar, March ??, 2019
- ?:00 PM to ?:00 PM
• Introduction to Azure IoT SuiteLocation:
• Microsoft Canada HQ in MississaugaLocation:
Future Events…
C# Corner ---
21. THANK YOU
Please feel free to survey
1. How did we/I do?
2. How can we improve?
3. What other public event are you interested?
Stay in touch:
SQLDataSide@gmail.com
http://sqldataside.blogspot.ca/
Sincerely
Nik
Thank You For Your Time