CSI – IT2020, IIT Mumbai, October 6th 2017
Computer Society of India, Mumbai Chapter
The presentation focuses on Microservices architecture and the comparison between MicroService with Standard Monolithic Apps and SOA based Apps. It also gives a quick outline of Domain Driven Design, Event Sourcing and CQRS, Functional Reactive Programming and comparison of SAGA pattern with 2 Phase Commit.
http://www.csimumbai.org/it2020-17/index.html
A microservice approach for legacy modernisationluisw19
Very large portion of the world’s business critical systems are considered to be ‘legacy’ –and so is the code underpinning them (ie COBOL, PASCAL, C, to name a few). Although in many cases it is the case that these systems are robust, stable and fit for the main purpose they were originally built, they aren’t flexible and scalable enough to support emerging requirements mainly derived from a more demanding ‘always on the move’ and ‘always connected’ user.
These systems struggle to meet these demands mainly because of the ‘monolithic’ approach on which they were built and the complexity hidden in millions of lines of code that is only understood a very few hand-full of people that still remain active from the teams that several years ago developed these systems.
In almost an equal amount there have also been thousands of failed attempts to modernise these legacy systems. The ‘eating the elephant’ in one go approach certainly didn’t work, and the traditional SOA approach alone although worked till certain extend, it also fell short when it came down to addressing specific requirements around scalability and platform/service inter-dependencies.
In this presentation I will talk about how a legacy modernisation framework based on Microservice Architecture (MSA) in conjunction with some other known SOA patterns (ie. ESB, API Gateway), can be applied to ‘eat the elephant one piece at the time’ but most importantly ‘without getting indigestion’
SCS 4120 - Software Engineering IV
BACHELOR OF SCIENCE HONOURS IN COMPUTER SCIENCE
BACHELOR OF SCIENCE HONOURS IN SOFTWARE ENGINEERING
All in One Place Lecture Notes
Distribution Among Friends Only
All copyrights belong to their respective owners
Viraj Brian Wijesuriya
vbw@ucsc.cmb.ac.lk
Kevin Huang: AWS San Francisco Startup Day, 9/7/17
Architecture: When, how, and if to adopt microservices - Microservices are not for everyone! If you're a small shop, a monolith provides a great amount of value and reduces the complexities involved. However as your company grows, this monolith becomes more difficult to maintain. We’ll look at how microservices allow you to easily deploy and debug atomic pieces of infrastructure which allows for increased velocity in reliable, tested, and consistent deploys. We’ll look into key metrics you can use to identify the right time to begin the transition from monolith to microservices.
CSI – IT2020, IIT Mumbai, October 6th 2017
Computer Society of India, Mumbai Chapter
The presentation focuses on Microservices architecture and the comparison between MicroService with Standard Monolithic Apps and SOA based Apps. It also gives a quick outline of Domain Driven Design, Event Sourcing and CQRS, Functional Reactive Programming and comparison of SAGA pattern with 2 Phase Commit.
http://www.csimumbai.org/it2020-17/index.html
A microservice approach for legacy modernisationluisw19
Very large portion of the world’s business critical systems are considered to be ‘legacy’ –and so is the code underpinning them (ie COBOL, PASCAL, C, to name a few). Although in many cases it is the case that these systems are robust, stable and fit for the main purpose they were originally built, they aren’t flexible and scalable enough to support emerging requirements mainly derived from a more demanding ‘always on the move’ and ‘always connected’ user.
These systems struggle to meet these demands mainly because of the ‘monolithic’ approach on which they were built and the complexity hidden in millions of lines of code that is only understood a very few hand-full of people that still remain active from the teams that several years ago developed these systems.
In almost an equal amount there have also been thousands of failed attempts to modernise these legacy systems. The ‘eating the elephant’ in one go approach certainly didn’t work, and the traditional SOA approach alone although worked till certain extend, it also fell short when it came down to addressing specific requirements around scalability and platform/service inter-dependencies.
In this presentation I will talk about how a legacy modernisation framework based on Microservice Architecture (MSA) in conjunction with some other known SOA patterns (ie. ESB, API Gateway), can be applied to ‘eat the elephant one piece at the time’ but most importantly ‘without getting indigestion’
SCS 4120 - Software Engineering IV
BACHELOR OF SCIENCE HONOURS IN COMPUTER SCIENCE
BACHELOR OF SCIENCE HONOURS IN SOFTWARE ENGINEERING
All in One Place Lecture Notes
Distribution Among Friends Only
All copyrights belong to their respective owners
Viraj Brian Wijesuriya
vbw@ucsc.cmb.ac.lk
Kevin Huang: AWS San Francisco Startup Day, 9/7/17
Architecture: When, how, and if to adopt microservices - Microservices are not for everyone! If you're a small shop, a monolith provides a great amount of value and reduces the complexities involved. However as your company grows, this monolith becomes more difficult to maintain. We’ll look at how microservices allow you to easily deploy and debug atomic pieces of infrastructure which allows for increased velocity in reliable, tested, and consistent deploys. We’ll look into key metrics you can use to identify the right time to begin the transition from monolith to microservices.
Integration Patterns for Microservices ArchitecturesNATS
NATS was created by Derek Collison, founder and CEO
of Apcera, who has spent 20+ years designing, building, and using publish-subscribe messaging systems.
Unlike traditional enterprise messaging systems, NATS has an always-on dial tone that does whatever it takes to remain available. Learn how end users are building modern, reliable and scalable cloud and distributed systems with NATS.
Talk given by David Williams, Principal, Williams & Garcia
You can learn more about NATS at http://www.nats.io
Entity Framework Core session on .NET core workshop. This slides contains the presentation on getting started on with Entity Framework Core 2.2. This was presented on Workshop On .Net
event held at Academia International College Tyagal - Lalitpur, Nepal. https://www.facebook.com/events/507837526666020/
And codes on the workshop is available on github https://github.com/kiranshahi/Todo-App
Here’s a step-by-step guide to implement Flask JWT Authentication with an example. Clone the flask-jwt authentication github repo and play around with the code
YouTube Link: https://youtu.be/h8uM4mezyHU
** DevOps Certification Courses - https://www.edureka.co/devops-certification-courses** This Edureka PPT on ‘DevOps Real-Time Scenarios’ will discuss the various real-time Challenges that you encounter while adopting or implementing DevOps practices.
Follow us to never miss an update in the future.
YouTube: https://www.youtube.com/user/edurekaIN
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
Castbox: https://castbox.fm/networks/505?country=in
Recently I was asked to explain what dev-ops is at a large enterprise software vendor undergoing transformation.
In these slides, I present the concepts, tools and mindset that drive DevOPS.
The presentation from our online webinar "Design patterns for microservice architecture".
Full video from webinar available here: https://www.youtube.com/watch?v=826aAmG06KM
If you’re a CTO or a Lead Developer and you’re planning to design service-oriented architecture, it’s definitely a webinar tailored to your needs. Adrian Zmenda, our Lead Dev, will explain:
- when microservice architecture is a safe bet and what are some good alternatives
- what are the pros and cons of the most popular design patterns (API Gateway, Backend for Frontend and more)
- how to ensure that the communication between services is done right and what to do in case of connection issues
- why we’ve decided to use a monorepo (monolithic repository)
- what we’ve learned from using the remote procedure call framework gRPC
- how to monitor the efficiency of individual services and whole SOA-based systems.
This presentation by Serhii Abanichev (System Architect, Consultant, GlobalLogic) was delivered at GlobalLogic Kharkiv DevOps TechTalk #1 on October 8, 2019.
In this talk were covered:
- Full coverage of DevOps with Azure DevOps Services:
- Create, test and deploy in any programming language, to any cloud or local environment.
- Run concurrently on Linux, macOS, and Windows, deploying containers for individual hosts or Kubernetes.
- Azure DevOps Services: a Microsoft solution that replaces dozens of tools ensuring smooth delivery to end users.
Event materials: https://www.globallogic.com/ua/events/kharkiv-devops-techtalk-1/
Introduction To DevOps | Devops Tutorial For Beginners | DevOps Training For ...Simplilearn
This presentation on "Introduction to DevOps" will help you understand what is waterfall model, what is an agile model, what is DevOps, DevOps phases, DevOps tools and DevOps advantages. In traditional software development lifecycle, there is a lot of gap between development and operations team. DevOps addresses the gap between developers and operations. The development team will submit the application to the operations team for implementation. Operations team will monitor the application and provide relevant feedback to developers. According to DevOps practices, the workflow in software development and delivery is divided into 8 phases, Now, let us get started and understand these 8 phases in DevOps.
Below topics are explained in this "Introduction to DevOps" presentation:
1. Waterfall model
2. Agile model
3. What is DevOps?
4. DevOps phases
5. DevOps tools
6. DevOps advantages
Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The Devops training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
Who should take this course?
DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461. DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
1. This DevOps training course will be of benefit the following professional roles:
2. Software Developers
3. Technical Project Managers
4. Architects
5. Operations Support
6. Deployment engineers
7. IT managers
8. Development managers
Learn more at: https://www.simplilearn.com/
An introduction to REST and RESTful web services.
You can take the course below to learn about REST & RESTful web services.
https://www.udemy.com/building-php-restful-web-services/
This is a talk I gave at PLoP 2017 - http://www.hillside.net/plop/2017/index.php?nav=program
The microservice architecture is growing in popularity. It is an architectural style that structures an application as a set of loosely coupled services that are organized around business capabilities. Its goal is to enable the continuous delivery of large, complex applications. However, the microservice architecture is not a silver bullet and it has some significant drawbacks.
The goal of the microservices pattern language is to enable software developers to apply the microservice architecture effectively. It is a collection of patterns that solve architecture, design, development and operational problems. In this talk, I’ll provide an overview of the microservice architecture and describe the motivations for the pattern language. You will learn about the key patterns in the pattern language.
Integration Patterns for Microservices ArchitecturesNATS
NATS was created by Derek Collison, founder and CEO
of Apcera, who has spent 20+ years designing, building, and using publish-subscribe messaging systems.
Unlike traditional enterprise messaging systems, NATS has an always-on dial tone that does whatever it takes to remain available. Learn how end users are building modern, reliable and scalable cloud and distributed systems with NATS.
Talk given by David Williams, Principal, Williams & Garcia
You can learn more about NATS at http://www.nats.io
Entity Framework Core session on .NET core workshop. This slides contains the presentation on getting started on with Entity Framework Core 2.2. This was presented on Workshop On .Net
event held at Academia International College Tyagal - Lalitpur, Nepal. https://www.facebook.com/events/507837526666020/
And codes on the workshop is available on github https://github.com/kiranshahi/Todo-App
Here’s a step-by-step guide to implement Flask JWT Authentication with an example. Clone the flask-jwt authentication github repo and play around with the code
YouTube Link: https://youtu.be/h8uM4mezyHU
** DevOps Certification Courses - https://www.edureka.co/devops-certification-courses** This Edureka PPT on ‘DevOps Real-Time Scenarios’ will discuss the various real-time Challenges that you encounter while adopting or implementing DevOps practices.
Follow us to never miss an update in the future.
YouTube: https://www.youtube.com/user/edurekaIN
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
Castbox: https://castbox.fm/networks/505?country=in
Recently I was asked to explain what dev-ops is at a large enterprise software vendor undergoing transformation.
In these slides, I present the concepts, tools and mindset that drive DevOPS.
The presentation from our online webinar "Design patterns for microservice architecture".
Full video from webinar available here: https://www.youtube.com/watch?v=826aAmG06KM
If you’re a CTO or a Lead Developer and you’re planning to design service-oriented architecture, it’s definitely a webinar tailored to your needs. Adrian Zmenda, our Lead Dev, will explain:
- when microservice architecture is a safe bet and what are some good alternatives
- what are the pros and cons of the most popular design patterns (API Gateway, Backend for Frontend and more)
- how to ensure that the communication between services is done right and what to do in case of connection issues
- why we’ve decided to use a monorepo (monolithic repository)
- what we’ve learned from using the remote procedure call framework gRPC
- how to monitor the efficiency of individual services and whole SOA-based systems.
This presentation by Serhii Abanichev (System Architect, Consultant, GlobalLogic) was delivered at GlobalLogic Kharkiv DevOps TechTalk #1 on October 8, 2019.
In this talk were covered:
- Full coverage of DevOps with Azure DevOps Services:
- Create, test and deploy in any programming language, to any cloud or local environment.
- Run concurrently on Linux, macOS, and Windows, deploying containers for individual hosts or Kubernetes.
- Azure DevOps Services: a Microsoft solution that replaces dozens of tools ensuring smooth delivery to end users.
Event materials: https://www.globallogic.com/ua/events/kharkiv-devops-techtalk-1/
Introduction To DevOps | Devops Tutorial For Beginners | DevOps Training For ...Simplilearn
This presentation on "Introduction to DevOps" will help you understand what is waterfall model, what is an agile model, what is DevOps, DevOps phases, DevOps tools and DevOps advantages. In traditional software development lifecycle, there is a lot of gap between development and operations team. DevOps addresses the gap between developers and operations. The development team will submit the application to the operations team for implementation. Operations team will monitor the application and provide relevant feedback to developers. According to DevOps practices, the workflow in software development and delivery is divided into 8 phases, Now, let us get started and understand these 8 phases in DevOps.
Below topics are explained in this "Introduction to DevOps" presentation:
1. Waterfall model
2. Agile model
3. What is DevOps?
4. DevOps phases
5. DevOps tools
6. DevOps advantages
Simplilearn's DevOps Certification Training Course will prepare you for a career in DevOps, the fast-growing field that bridges the gap between software developers and operations. You’ll become an expert in the principles of continuous development and deployment, automation of configuration management, inter-team collaboration and IT service agility, using modern DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios. DevOps jobs are highly paid and in great demand, so start on your path today.
Why learn DevOps?
Simplilearn’s DevOps training course is designed to help you become a DevOps practitioner and apply the latest in DevOps methodology to automate your software development lifecycle right out of the class. You will master configuration management; continuous integration deployment, delivery and monitoring using DevOps tools such as Git, Docker, Jenkins, Puppet and Nagios in a practical, hands-on and interactive approach. The Devops training course focuses heavily on the use of Docker containers, a technology that is revolutionizing the way apps are deployed in the cloud today and is a critical skillset to master in the cloud age.
Who should take this course?
DevOps career opportunities are thriving worldwide. DevOps was featured as one of the 11 best jobs in America for 2017, according to CBS News, and data from Payscale.com shows that DevOps Managers earn as much as $122,234 per year, with DevOps engineers making as much as $151,461. DevOps jobs are the third-highest tech role ranked by employer demand on Indeed.com but have the second-highest talent deficit.
1. This DevOps training course will be of benefit the following professional roles:
2. Software Developers
3. Technical Project Managers
4. Architects
5. Operations Support
6. Deployment engineers
7. IT managers
8. Development managers
Learn more at: https://www.simplilearn.com/
An introduction to REST and RESTful web services.
You can take the course below to learn about REST & RESTful web services.
https://www.udemy.com/building-php-restful-web-services/
This is a talk I gave at PLoP 2017 - http://www.hillside.net/plop/2017/index.php?nav=program
The microservice architecture is growing in popularity. It is an architectural style that structures an application as a set of loosely coupled services that are organized around business capabilities. Its goal is to enable the continuous delivery of large, complex applications. However, the microservice architecture is not a silver bullet and it has some significant drawbacks.
The goal of the microservices pattern language is to enable software developers to apply the microservice architecture effectively. It is a collection of patterns that solve architecture, design, development and operational problems. In this talk, I’ll provide an overview of the microservice architecture and describe the motivations for the pattern language. You will learn about the key patterns in the pattern language.
This session was done during the SQL Saturdays #115 held in Portugal and provides a first look to the new SQL Integration Services (SSIS) version. For more details visit my blog at www.pedrocgd.blogspot.com
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours o...Cathrine Wilhelmsen
Don't Repeat Yourself - An Introduction to Agile SSIS Development (24 Hours of PASS) (Presented at 24 HOurs of PASS Growing Our Community Edititon on June 25th 2015)
EnterJS 2015 - Continuous Integration for Frontend CodeMarcel Birkner
Continuous Integration gehört in den meisten Unternehmen mittlerweile zum guten Ton, oft aber nur mit Blick auf die Server-Seite. Dabei kommt heute keine Anwendung mehr ohne Frontend-Code aus. Dieser wird - gerade im Enterprise-Bereich - oft vernachlässigt. In diesem Vortrag wird ein CI-Setup mit Fokus auf das Frontend vorgestellt. Automatisierte Tests, Sourcecode Quality Management und andere typische Bestandteile sind für den Frontend-Code genauso wichtig wie für das Backend. Durch Asset-Optimierung kann die Performance, gerade für mobile Clients, deutlich gesteigert werden. Gezeigt werden Tools, Methoden und Best Practices für den Aufbau und die Umsetzung der Delivery-Pipeline. Zum Einsatz kommen dabei Grunt, ESLint, Mocha und Jenkins.
Similar to Biml for Beginners - Generating SSIS Packages with BimlScript (SQLSaturday Exeter) (20)
The Battle of the Data Transformation Tools (PASS Data Community Summit 2023)Cathrine Wilhelmsen
The Battle of the Data Transformation Tools (Presented as part of the "Batte of the Data Transformation Tools" Learning Path at PASS Data Community Summit on November 16th, 2023)
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (PAS...Cathrine Wilhelmsen
Visually Transform Data in Azure Data Factory or Azure Synapse Analytics (Presented as part of the "Batte of the Data Transformation Tools" Learning Path at PASS Data Community Summit on November 15th, 2023)
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power ...Cathrine Wilhelmsen
Building an End-to-End Solution in Microsoft Fabric: From Dataverse to Power BI (Presented at SQLSaturday Oregon & SW Washington on November 11th, 2023)
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (S...Cathrine Wilhelmsen
Stressed, Depressed, or Burned Out? The Warning Signs You Shouldn't Ignore (Presented at SQLBits on March 18th, 2023)
We all experience stress in our lives. When the stress is time-limited and manageable, it can be positive and productive. This kind of stress can help you get things done and lead to personal growth. However, when the stress stretches out over longer periods of time and we are unable to manage it, it can be negative and debilitating. This kind of stress can affect your mental health as well as your physical health, and increase the risk of depression and burnout.
The tricky part is that both depression and burnout can hit you hard without the warning signs you might recognize from stress. Where stress barges through your door and yells "hey, it's me!", depression and burnout can silently sneak in and gradually make adjustments until one day you turn around and see them smiling while realizing that you no longer recognize your house. I know, because I've dealt with both. And when I thought I had kicked them out, they both came back for new visits.
I don't have the Answers™️ or Solutions™️ to how to keep them away forever. But in hindsight, there were plenty of warning signs I missed, ignored, or was oblivious to at the time. In this deeply personal session, I will share my story of dealing with both depression and burnout. What were the warning signs? Why did I miss them? Could I have done something differently? And most importantly, what can I - and you - do to help ourselves or our loved ones if we notice that something is not quite right?
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced ...Cathrine Wilhelmsen
"I can't keep up!" - Turning Discomfort into Personal Growth in a Fast-Paced World (Presented at SQLBits on March 17th, 2023)
Do you sometimes think the world is moving so fast that you're struggling to keep up?
Does it make you feel a little uncomfortable?
Awesome!
That means that you have ambitions. You want to learn new things, take that next step in your career, achieve your goals. You can do anything if you set your mind to it.
It just might not be easy.
All growth requires some discomfort. You need to manage and balance that discomfort, find a way to push yourself a little bit every day without feeling overwhelmed. In a fast-paced world, you need to know how to break down your goals into smaller chunks, how to prioritize, and how to optimize your learning.
Are you ready to turn your "I can't keep up" into "I can't believe I did all of that in just one year"?
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing S...Cathrine Wilhelmsen
Lessons Learned: Implementing Azure Synapse Analytics in a Rapidly-Changing Startup (Presented at SQLBits on March 11th, 2022)
What happens when you mix one rapidly-changing startup, one data analyst, one data engineer, and one hypothesis that Azure Synapse Analytics could be the right tool of choice for gaining business insights?
We had no idea, but we gave it a go!
Our ambition was to think big, start small, and act fast – to deliver business value early and often.
Did we succeed?
Join us for an honest conversation about why we decided to implement Azure Synapse Analytics alongside Power BI, how we got started, which areas we completely messed up at first, what our current solution looks like, the lessons learned along the way, and the things we would have done differently if we could start all over again.
6 Tips for Building Confidence as a Public Speaker (SQLBits 2022)Cathrine Wilhelmsen
6 Tips for Building Confidence as a Public Speaker (Presented at SQLBits on March 10th, 2022)
Do you feel nervous about getting on stage to deliver a presentation?
That was me a few years ago. Palms sweating. Hands shaking. Voice trembling. I could barely breathe and talked at what felt like a thousand words per second. Now, public speaking is one of my favorite hobbies. Sometimes, I even plan my vacations around events! What changed?
There are no shortcuts to building confidence as a public speaker. However, there are many things you can do to make the journey a little easier for yourself. In this session, I share the top tips I have learned over the years. All it takes is a little preparation and practice.
You can do this!
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Discussion on Vector Databases, Unstructured Data and AI
https://www.meetup.com/unstructured-data-meetup-new-york/
This meetup is for people working in unstructured data. Speakers will come present about related topics such as vector databases, LLMs, and managing data at scale. The intended audience of this group includes roles like machine learning engineers, data scientists, data engineers, software engineers, and PMs.This meetup was formerly Milvus Meetup, and is sponsored by Zilliz maintainers of Milvus.
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
Analysis insight about a Flyball dog competition team's performanceroli9797
Insight of my analysis about a Flyball dog competition team's last year performance. Find more: https://github.com/rolandnagy-ds/flyball_race_analysis/tree/main
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...sameer shah
"Join us for STATATHON, a dynamic 2-day event dedicated to exploring statistical knowledge and its real-world applications. From theory to practice, participants engage in intensive learning sessions, workshops, and challenges, fostering a deeper understanding of statistical methodologies and their significance in various fields."
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
4. Session description
SSIS is a powerful tool for extracting, transforming and loading data, but
creating the actual SSIS packages can be both tedious and time-
consuming. Even if you use templates and follow best practices you
often have to repeat the same steps over and over again. There are no
easy ways to handle metadata and schema changes, and if there are
new requirements you might have to go through all the packages one
more time. It's time to bring the Don't Repeat Yourself principle to SSIS
development.
In this session I will use the free BIDS Helper add-in to show you the
basics of Biml and BimlScript, how to generate SSIS packages
automatically from databases, how easy those packages can be
changed, and how to move common code to separate files that can be
included where needed. See why they say Biml allows you to complete
in a day what once took more than a week!
9. How can Biml help you?
Timesaving: Many SSIS
Packages from one Biml file
Reusable: Write once and run
on any platform (2005 – 2014)
Flexible: Start simple, expand
as you learn
(Of course I can create 200 packages!
What do you need me to do after lunch?)
10. Business Intelligence Markup Language
Easy to read and write XML dialect
Specifies business intelligence objects
Databases, schemas, tables, columns
SSIS packages
SSAS cubes, facts, dimensions (Mist only)
11. Highlights in Biml History
founded by Scott Currie, is born
Biml was extended with
Biml compiler added to
is launched
founded
is launched
2008:
2009:
2011:
2012:
2014:
2015:
20. Getting started with Biml
1. Download and install BIDS Helper (http://bidshelper.codeplex.com)
2. Right-click on SSIS project and click Add New Biml File
31. The magic is in the
Extend Biml with C# or VB.NET code blocks
Import database structure and metadata
Loop over tables and columns
Add expressions to replace static values
(And anything else you can do in C# or VB)
39. foreach (table in a database) loop
<#@ import namespace="Varigence.Hadron.CoreLowerer.SchemaManagement" #>
<# var conAW2014 = SchemaManager.CreateConnectionNode("AW2014", "Data Source...");
#>
<# var AW2014DB = conAW2014.ImportDB("","", ImportOptions.ExcludeViews); #>
<Packages>
<# foreach (var table in AW2014DB.TableNodes) { #>
<Package Name="Load_<#=table.Schema#>_<#=table.Name#>">
</Package>
<# } #>
</Packages>
40. Don't Repeat Yourself
Move common code to separate files
Centralize and reuse in many projects
Update code once for all projects
1. Split and combine Biml files
2. Include files
3. CallBimlScript with parameters
41. Split and combine Biml files
Multiple Biml files can be compiled together
Control compile order by specifying tiers in files
<#@ template tier="2" #>
Files are compiled into RootNode from lowest to highest tier
Higher tiers can use objects in RootNode from lower tiers
42. Behind the scenes: compile and load objects into RootNode
RootNode
<#@ template tier="0" #>
<Connections>
<Databases>
<Schemas>
<#@ template tier="1" #>
<Tables>
<Columns>
<#@ template tier="2" #>
<Packages>
52. Split and combine multiple Biml files
Select all the tiered files
Right-click and click Generate SSIS
Packages
Behind the scenes: Objects will be
compiled and loaded into RootNode
from lowest to highest tier
53. Split and combine multiple Biml files
All packages will be generated at the same time
Load packages from 302LoadAllTables.biml
Master package from 303MasterPackage.biml
54. Include files
Include common code in multiple files and projects
Use the include directive
<#@ include file="CommonCode.biml" #>
Include directive will be replaced by content of file
Can include several file types: .biml .txt .sql .cs
56. CallBimlScript with parameters
Works like a parameterized include
File to be called (callee) specifies input parameters
<#@ property name="Param" type="String" #>
Callee can use parameter values as regular variables and to
control logic
File that calls (caller) provides input parameters
<#=CallBimlScript("CommonCode.biml", Param)#>
CallBimlScript code block is replaced by Biml returned by callee
59. View compiled Biml
Credits: Marco Schreuder (@in2bi)
http://blog.in2bi.eu/biml/viewing-or-saving-the-
compiled-biml-file-s/
Helper file with high tier (tier="100")
Saves output of RootNode.GetBiml() to file
60. What do you do next?
1. Download BIDS Helper
2. Identify your SSIS patterns
3. Rewrite one SSIS package to Biml to learn the basics
4. Expand with BimlScript
5. Get involved in the Biml community