Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A "M"ind Bending Experience. Power Query for Power BI and Beyond.

480 views

Published on

Power Query is quickly becoming the number one skill your employees NEED TO HAVE working in a modern workplace. Join Alex Powers as he demystifies the Power Query M Language, writing your own custom solutions and how to approach the language in a way that makes sense to both technical and non-technical users.

Published in: Technology
  • Login to see the comments

A "M"ind Bending Experience. Power Query for Power BI and Beyond.

  1. 1. Power Query M? (Informally known as "M") The Power Query Formula Language is a powerful query language optimized for building queries that mashup data. It's a functional, case sensitive language. For computer language theorists: Power Query is a mostly pure, higher-order, dynamically typed, partially lazy, functional language. You can create a variety of data mashup queries from simple to advanced scenarios. (Source: MSDN)
  2. 2. Power Query?! (Data Connectivity and Preparation) Power Query enables business users to seamlessly access data stored in hundreds of data sources and reshape it to fit their needs, with an easy to use, engaging and no-code user experience. Data Connectors: (Source: Office Support) Access Database2 Hadoop File (HDFS) Microsoft Azure Table Storage PostgreSQL Database Text Active Directory IBM DB2 Database Microsoft Exchange SharePoint List CSV Excel Microsoft Azure HDInsight MySQL Database SQL Server Database Web Page Facebook Microsoft Azure Marketplace OData Feed Sybase Database XML Folder Microsoft Azure SQL Database Oracle Database Teradata Database And More…
  3. 3. “M” POWER QUERY FOR POWER BI
  4. 4. Alex M PowersPremier Field Engineer – Data & AI @ Microsoft Co-Organizer St Louis Power BI User Group #STLPBIUG Professional: • Microsoft Certified Solutions Expert: Data Management and Analytics • Microsoft Certified Solutions Associate: BI Reporting (Excel, Power BI) • Microsoft Certified Associate: Azure AI Engineer • Microsoft Certified Trainer • Microsoft Technology Associate: Python • Microsoft Office Expert & Specialist: Excel, Access • Microsoft #HowWeExcel Contest Winner Personal: • Met the Wife on Craigslist • Enjoys Pineapple on his Pizza
  5. 5. Click to edit master title style SESSION OBJECTIVES • Convince you why Power Query (M) should be on every resume. • Have fun.
  6. 6. MAKE IT FEEL FAMILIAR. BEFORE IT FEELS FOREIGN.
  7. 7. Click to edit master title style PRIMITIVE VALUES Type Example binary #binary("AQID") date #date(2018,7,10) time #time(18,30,0) datetime #datetime(2018,7,10,18,30,0) datetimezone #datetimezone(2018,7,10,18,30,0,0,6) duration #duration(1,0,0,0) logical true and false null null number 0, 1, -1, 1.5 text “hello, world”
  8. 8. Click to edit master title style STRUCTURED VALUES Type Example list {“foo”, “bar”, 10} record [a = 10, b = 20] table #table({“Animal”}, {{“Cat”},{“Dog”}})
  9. 9. let Source = “Hello, World!” in Source Power Query M’s let expression stores a set of named values called variables. Each variable has a unique name called an identifier. In the above let expression – the variable Source stores the primitive text value “Hello, World!”. The final expression is evaluated in the identifier titled Source. let in let Source = in let Source = “Hello, World!” in
  10. 10. let Source = "Hello, World", #"Something Else" = "Keepin' It", Val = 100, #"Struct List" = {Source, #"Something Else", Val}, #"List – Index" = #"Struct List"{2}, // 0 Based Index #"Numbered List - 0 to 100" = {0..#"List – Index"}, #"Table With Meta" = Table.FromList(#"Numbered List - 0 to 100", Splitter.SplitByNothing(), type table [Index = Int64.Type], null, ExtraValues.Error), #"Each Loop with ASCII Number" = Table.AddColumn(#"Table With Meta", "ASCII", each Character.FromNumber([Index]), type text), #"ASCII Column and Row" = #"Each Loop with ASCII Number"[ASCII]{97}, #"Alphabetical List - a to z" = {#"ASCII Column and Row".."z"} in #"Alphabetical List - a to z" In the above let expression – the identifiers enclosed in #” “, denote the use of a special character (space) or number for the variables identifier. Starting from the beginning of the expression we begin with primitive values, introduce lists { }, positional index operator {2}, list sequences for numbers {0..100}, for each list item additional column transformations, lookup and positional index operators [Column1]{2} and the use of alphabetical list sequences {“a”..”z”}. Full Solution: Familiar Feeling
  11. 11. To view the available Power Query M functions use the formula =#shared to view as structured records or =Record.ToTable(#shared) to cast the records into a structured table format. =#shared
  12. 12. WE CAN REBUILD IT. WE HAVE THE TECHNOLOGY. BETTER THAN IT WAS BEFORE. BETTER. STRONGER. FASTER. THE SIX MILLION DOLLAR QUERY.
  13. 13. let Source = Excel.Workbook(Web.Contents("https://www.ers.usda.gov/webdocs/DataFiles/48747/Unemployment.xls"), null, true), #"Unemployment Med HH Inc_Sheet" = Source{[Name="Unemployment Med HH Inc"]}[Data], // Extract the previous identifier's Column1 as a list. Find the position of the text "FIPStxt", its first occurence while ignoring case sensitivity #"Find Position" = Table.Skip( #"Unemployment Med HH Inc_Sheet", List.PositionOf( #"Unemployment Med HH Inc_Sheet"[Column1], "FIPStxt", 1, Comparer.OrdinalIgnoreCase ) ), #"Promoted Headers" = Table.PromoteHeaders(#"Find Position", [PromoteAllScalars=true]), /* Clean each item in Column Names. The use of a underscore avoids the hardcoding of individual column headers and instead transforms each item in the collection We used a Replacer to ReplaceText of underscores to spaces in our headers, this function is wrapped with a Text.Proper to standardize all column headers */ #"Clean Headers" = Table.TransformColumnNames( #"Promoted Headers", each Text.Proper( Replacer.ReplaceText( _ , "_", " ") ) ), #"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Clean Headers", {"Fipstxt", "State", "Area Name"}, "Attribute", "Value"), // Rename the auto generated Attribute.2 Column Header to Year. This step avoids and unnnecessary Table.RenameColumns function #"Split Column by Delimiter" = Table.SplitColumn(#"Unpivoted Other Columns", "Attribute", Splitter.SplitTextByEachDelimiter({" "}, QuoteStyle.Csv, true), {"Attribute.1", "Year"}), #"Pivoted Column" = Table.Pivot(#"Split Column by Delimiter", List.Distinct(#"Split Column by Delimiter"[Attribute.1]), "Attribute.1", "Value", List.Count), #"Changed Type" = Table.TransformColumnTypes(#"Pivoted Column",{{"Fipstxt", type text}, {"State", type text}, {"Area Name", type text}, {"Year", Int64.Type}, {"Civilian Labor Force", Int64.Type}, {"Employed", Int64.Type}, {"Unemployed", Int64.Type}, {"Unemployment Rate", Int64.Type}, {"Median Household Income", Int64.Type}, {"Med Hh Income Percent Of State Total", Int64.Type}, {"Rural Urban Continuum Code", Int64.Type}, {"Urban Influence Code", Int64.Type}, {"Metro", Int64.Type}}) in #"Changed Type" In the above let expression – we introduce future proofing the header row indicator of "FIPStxt“ in our dataset and ignoring case sensitivity. Transforming our column names with an each loop we utilize the underscore character to iterate thru each item in our column headers collection. Full Solution: Six Million Dollar Query
  14. 14. NoSQLNO. NOT THAT KIND.
  15. 15. Query Folding One of the most powerful capabilities of Power Query and the M Language is Query Folding (also referred to as query delegation, and predicate push-down). Query Folding allows the M Engine to push the transformations expressed in an M query to the source, in the source's native query language, resulting in more efficient data processing. Data sources will support different levels of query capabilities. To provide a consistent data transformation experience, the M engine compensates (i.e. does the processing locally) for transformations that cannot be sent to the source. It is the Data Connector's responsibility to report its capabilities to the engine, carving off the transformations it can handle, generating the appropriate query syntax, and letting the M Engine handle the remaining work. (Source: Microsoft Docs)
  16. 16. Full Solution: Query Folding PowerPlatformSummit Custom Column Name Transition Function: fxColumnNamesTransition
  17. 17. ROADMAPS?... WHERE WE’RE GOING, WE DON’T NEED ROADMAPS. THE FUTURE IS OURS TO WRITE.
  18. 18. IF YOU WANT YOUR SERVICE TO BE A HIT. MAKE SURE IT WORKS WITH EXCEL. Oct-11 SQL Azure Labs Feb-13 Excel Add-In Jul-13 Excel (Natively) Dec-14 Power BI Apr-17 Azure Analysis Services Mar-18 Flow Aug-17 PowerApps Nov-18 Dataflows Feb-19 SSIS May-19 Azure Data Factory May-19 Excel (Mac)
  19. 19. FLOW
  20. 20. PowerApps
  21. 21. Azure Analysis Services - 1400
  22. 22. Power BI service
  23. 23. VISUAL STUDIO
  24. 24. THE FUTURE M-DBA (Highly Specialized in Writing Optimized M Syntax and Building Custom Connectors) Intellisense (DONE) Seamless Integration Across All Offerings (Working Toward a Unified Experience) SSIS and Azure Data Factory Integration (DONE… or in Preview) Microsoft Access (WHY NOT?) Excel for Mac (DONE) Excel Online (Crystal Ball Prediction) Version Control (GitHub) Standalone Product (WHY NOT?)
  25. 25. RESOURCES Power Query Language Specification Guide https://docs.microsoft.com/en-us/powerquery-m/power-query-m-language-specification Collect, Combine, and Transform Data Using Power Query in Excel and Power BI https://www.amazon.com/Collect-Combine-Transform-Business-Skills/dp/1509307958 M Is for (Data) Monkey: A Guide to the M Language in Excel Power Query https://www.amazon.com/Data-Monkey-Guide-Language-Excel/dp/1615470344/ Excel 2016: Get & Transform Data (Power Query) https://www.lynda.com/Excel-tutorials/About-Get-Transform/608994/645967-4.html
  26. 26. CLOSING REMARKS IT’S NOT ABOUT WHAT YOU KNOW. IT’S ABOUT WHAT YOU KNOW IS POSSIBLE.
  27. 27. QUESTIONS? SUGGESTIONS AM I CRAZY?!
  28. 28. Click to edit master title style Thank you for attending! Please complete your survey
  29. 29. Alex Power[s] BI @notaboutthecell /in/alexmpowers

×