Invited talk at Document Intelligence workshop at KDD'2021.
Harvesting information from complex documents such as in financial reports and scientific publications is critical to building AI applications for business and research. Such documents are often in PDF format with critical facts and data conveyed in table and graphs. Extracting such information is essential to extract insights from these documents. In IBM Research, we have a rich agenda in this area that we call Deep Document Understanding. In this talk, I will focus on our research on Deep Table Understanding — extracting and understanding tables from PDF documents. I will introduce key challenges in table extraction and understanding and how we address such challenges, from how to acquire data at scale to enable deep neural network models to how to build, customize and evaluate such models. I will also describe how our work enables real-world use cases in domains such as finance and life science. Finally, I will briefly present TableQA, an important downstream task enabled by Deep Table Understanding.
This document brings together a set of latest data points and publicly available information relevant for Hybrid Cloud Infrastructure Industry. We are very excited to share this content and believe that readers will benefit from this periodic publication immensely.
1) PV & FV computations. 1. If you deposit $100,000 into a savi.docxjeremylockett77
1) PV & FV computations.
1. If you deposit $100,000 into a saving account yielding 8%, how big will it grow to in 5 years? Assume quarterly compounding.
2. What amount should be deposited in a bank account today to receive $300,000 in 6 years at 12 % interest rate? Assume quarterly compounding.
3. If $4,000 is deposited into an investment account yielding 10% every quarter starting on 1/1/2018, what amount will be available in the investment account in 4 years.
4. What is the present value of 5 $10,000 payments each of which will be received at the beginning of each period, discounted at 6% per a compounding period.
5. Hawk, Inc. issued 20,000 shares of bond on 1/1/2018. Those are $1,000/ share par value, 6 year, 6% stated interest rate bonds. The prevailing market rate on 1/1/2018 was 8%.
Calculate the price of the bond.
6. Hawk, Inc. considers the Pepsi Bottling Company and Coca Cola Bottling Company in
Salisbury for business expansion. Financial analysts specialized in valuation of soft drink companies estimated that Pepsi bottling will earn $400,000 a year over the next 25 years, while Coca-Cola Bottling will earn $600,000 over the next 30 years. Due to different credit rating of the two companies, the prevailing market interest rate of Pepsi is 10% and that of Coca-Cola is 8%. The Pepsi is asking $3,400,000, while The Coca-Cola is asking $6,000,000. Which is the better
offer to Hawk?
7. Hawk, Inc purchased a rental property for $1,000,000. This property will be leased for 10
years and the salvage value will be $400,000 after 10 years. How much annual rent should Hawk charge to earn 8% return on the investment?
2)
Company
Adjusted Trail Balance
As of 12/31/2018
AccountDebitCredit
Cash $30,000
Accounts receivable 40,000
Allowance for doubtful accounts $ 1,000
Notes receivable 18,000
Inventory,1/1/2018 51,000
Prepaid insurance 6,000
Long-term investment in MS stock 80,000
Buildings 500,000
Accumulated depreciated on buildings300,000
Furniture and equipment 110,000
Accumulated depreciation of F & E 30,000
Patent 90,000
Accounts payable 20,000
Bonds Payable100,000
Common stock 360,000
Retained earnings 257,000
Dividends 160,000
Sales 410,000
COGS 243,000
Salary expense 50,000
Rent expense 60,000
Gain on sale of discontinued operation 60,000*
Net loss from operations of
discontinued operation 100,000*
Totals 1,538,0001,538,000
*net of tax amounts, tax = 30%.
Instructions: using information on the above adjusted trial balance prepa ...
Sample Report: Global Digital Gaming Market 2017yStats.com
Free Report Samples for our publication "Global Digital Gaming Market 2017".
Find the full updated 2019 report available for purchase at: https://bit.ly/3N6YEjm
Recommended Strategies and Long-Term ObjectivesUpon review .docxdanas19
Recommended Strategies and Long-Term Objectives
Upon review of the data provided within the appendices, in conjunction with the substantive strategic analyses noted above, there seems to be a clear strategy for iRobot to take to gain a competitive advantage in the near future. Furthermore, this strategy will ensure the company’s financial security and exponential growth for the next decade. Within three years, iRobot will have fully absorbed the new strategy’s initial costs and will provide substantial increases in net income and cash flows, which in turn will result in impressive financial statements to appeal to investors, as well as improved operating efficiency within the company to allow it to expand to new markets.
iRobot has an increasing number of competitors within its market, and its current market share is relatively small, despite the company’s continual growth over the past several years. The company has put little effort into its marketing campaigns, and has also placed few resources to research and development. However, the recommended strategy for the company will be to use considerable capital in research and development, to create innovative robots designed for the retail industry. Major retail companies, such as Walmart, are beginning to invest in robots to facilitate a great number of tasks, both in physical retail locations, as well as manufacturing and distribution centers. With the commitment from companies such as these to continue integrating robotics into their operations, a new lucrative market is available for iRobot. If the company could develop a robot to facilitate the needs of these retail giants, iRobot could recognize massive profits, and also capitalize on relatively untouched market, quickly grabbing up the majority of the market share.
The suggested strategy is for iRobot to invest $70 million in 2019 in the R & D department, to design and produce a retail-specific robot within 6 months. The company currently has more than enough liquid assets to cover this investment without putting the company in financial stress. Once the robots are developed, iRobot will invest $25 million in a marketing campaign geared specifically for the retail industry, to gain the attention of retailers and supply chain companies worldwide. In 2019, iRobot will purchase 5,000 robots, with the goal of selling 2,500 in the first year. The average cost for such a robot will be around $15,500 per robot for production, while the average sales price that iRobot could charge to retailers is around $50,000 per robot.
For the first year, iRobot will incur and additional $172.5 million in the initial investment and production of the first run of 5,000 robots. However, the company will also recognize an additional $125 million in revenues from the 2,500 robots to be sold in 2019. This will result in a decrease in net income of around 44%, which still nets the company nearly $50 million in net income. Although the first year represents .
This document brings together a set of latest data points and publicly available information relevant for Hybrid Cloud Infrastructure Industry. We are very excited to share this content and believe that readers will benefit from this periodic publication immensely.
1) PV & FV computations. 1. If you deposit $100,000 into a savi.docxjeremylockett77
1) PV & FV computations.
1. If you deposit $100,000 into a saving account yielding 8%, how big will it grow to in 5 years? Assume quarterly compounding.
2. What amount should be deposited in a bank account today to receive $300,000 in 6 years at 12 % interest rate? Assume quarterly compounding.
3. If $4,000 is deposited into an investment account yielding 10% every quarter starting on 1/1/2018, what amount will be available in the investment account in 4 years.
4. What is the present value of 5 $10,000 payments each of which will be received at the beginning of each period, discounted at 6% per a compounding period.
5. Hawk, Inc. issued 20,000 shares of bond on 1/1/2018. Those are $1,000/ share par value, 6 year, 6% stated interest rate bonds. The prevailing market rate on 1/1/2018 was 8%.
Calculate the price of the bond.
6. Hawk, Inc. considers the Pepsi Bottling Company and Coca Cola Bottling Company in
Salisbury for business expansion. Financial analysts specialized in valuation of soft drink companies estimated that Pepsi bottling will earn $400,000 a year over the next 25 years, while Coca-Cola Bottling will earn $600,000 over the next 30 years. Due to different credit rating of the two companies, the prevailing market interest rate of Pepsi is 10% and that of Coca-Cola is 8%. The Pepsi is asking $3,400,000, while The Coca-Cola is asking $6,000,000. Which is the better
offer to Hawk?
7. Hawk, Inc purchased a rental property for $1,000,000. This property will be leased for 10
years and the salvage value will be $400,000 after 10 years. How much annual rent should Hawk charge to earn 8% return on the investment?
2)
Company
Adjusted Trail Balance
As of 12/31/2018
AccountDebitCredit
Cash $30,000
Accounts receivable 40,000
Allowance for doubtful accounts $ 1,000
Notes receivable 18,000
Inventory,1/1/2018 51,000
Prepaid insurance 6,000
Long-term investment in MS stock 80,000
Buildings 500,000
Accumulated depreciated on buildings300,000
Furniture and equipment 110,000
Accumulated depreciation of F & E 30,000
Patent 90,000
Accounts payable 20,000
Bonds Payable100,000
Common stock 360,000
Retained earnings 257,000
Dividends 160,000
Sales 410,000
COGS 243,000
Salary expense 50,000
Rent expense 60,000
Gain on sale of discontinued operation 60,000*
Net loss from operations of
discontinued operation 100,000*
Totals 1,538,0001,538,000
*net of tax amounts, tax = 30%.
Instructions: using information on the above adjusted trial balance prepa ...
Sample Report: Global Digital Gaming Market 2017yStats.com
Free Report Samples for our publication "Global Digital Gaming Market 2017".
Find the full updated 2019 report available for purchase at: https://bit.ly/3N6YEjm
Recommended Strategies and Long-Term ObjectivesUpon review .docxdanas19
Recommended Strategies and Long-Term Objectives
Upon review of the data provided within the appendices, in conjunction with the substantive strategic analyses noted above, there seems to be a clear strategy for iRobot to take to gain a competitive advantage in the near future. Furthermore, this strategy will ensure the company’s financial security and exponential growth for the next decade. Within three years, iRobot will have fully absorbed the new strategy’s initial costs and will provide substantial increases in net income and cash flows, which in turn will result in impressive financial statements to appeal to investors, as well as improved operating efficiency within the company to allow it to expand to new markets.
iRobot has an increasing number of competitors within its market, and its current market share is relatively small, despite the company’s continual growth over the past several years. The company has put little effort into its marketing campaigns, and has also placed few resources to research and development. However, the recommended strategy for the company will be to use considerable capital in research and development, to create innovative robots designed for the retail industry. Major retail companies, such as Walmart, are beginning to invest in robots to facilitate a great number of tasks, both in physical retail locations, as well as manufacturing and distribution centers. With the commitment from companies such as these to continue integrating robotics into their operations, a new lucrative market is available for iRobot. If the company could develop a robot to facilitate the needs of these retail giants, iRobot could recognize massive profits, and also capitalize on relatively untouched market, quickly grabbing up the majority of the market share.
The suggested strategy is for iRobot to invest $70 million in 2019 in the R & D department, to design and produce a retail-specific robot within 6 months. The company currently has more than enough liquid assets to cover this investment without putting the company in financial stress. Once the robots are developed, iRobot will invest $25 million in a marketing campaign geared specifically for the retail industry, to gain the attention of retailers and supply chain companies worldwide. In 2019, iRobot will purchase 5,000 robots, with the goal of selling 2,500 in the first year. The average cost for such a robot will be around $15,500 per robot for production, while the average sales price that iRobot could charge to retailers is around $50,000 per robot.
For the first year, iRobot will incur and additional $172.5 million in the initial investment and production of the first run of 5,000 robots. However, the company will also recognize an additional $125 million in revenues from the 2,500 robots to be sold in 2019. This will result in a decrease in net income of around 44%, which still nets the company nearly $50 million in net income. Although the first year represents .
This document brings together a set
of latest data points and publicly
available information relevant for
Business Services Industry. We are
very excited to share this content and
believe that readers will benefit from
this periodic publication immensely.
This document brings together a set of latest data points and publicly available information relevant for Business Services Industry. We are very excited to share this content and believe that readers will benefit immensely from this periodic publication immensely.
Sample Report: Global Online Gaming Market 2015yStats.com
Free Report Samples for our publication "Global Online Gaming Market 2015".
Find the full updated 2019 report available for purchase at: https://ystats.com/shop/global-digital-gaming-market-2019-2/
The “Gartner Perspective: IT Spending” booklet provides an
overview of Gartner research on IT spending and functions as
a reference guide to top-level statistics and IT spending analysis.
It provides a glimpse into the
powerful insight Gartner can
provide as you navigate through
what may be the most important
year of your career.
Rong Viet Securities - Investment Strategy June 2018Thomas Farthofer
In their recently published strategy report for June 2018, our partner Rong Viet explains why, despite a severe correction in the stock market during April and May, investors are in no need to rush in to buy massively. Still, valuations now appear more reasonable and it seems that it is time to gradually accumulate stocks.
Access to this presentation has been made possible through "Sao Bien. Room for Education", an Austrian-based non-profit organization and cooperation partner of Viet Dragon Securities.
Reprinted with the permission of Viet Dragon Securities. Not for US investors.
Sample Report: Europe Online Gaming Market 2015yStats.com
Free Report Samples for our publication "Europe Online Gaming Market 2015".
Find the full updated "Global Digital Gaming Market 2019" report available for purchase at: https://ystats.com/shop/global-digital-gaming-market-2019-2/
This document brings together a set of latest data points and publicly available information relevant for Agile & AI Operations. We are very excited to share this content and believe that readers will benefit immensely from this periodic publication immensely.
The Role of Patterns in the Era of Large Language ModelsYunyao Li
Slides for my keynote at PAN-DL Workshop (Pattern-based Approaches to NLP in the Age of Deep Learning) at EMNLP'2023 (December. 6, 2023).
In this talk, I share our initial learnings from constructing, growing and serving large knowledge graphs
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-LoopYunyao Li
Keynote talk at HILDA'2023 at SIGMOD on June 18, 2023.
Abstract: The ability to build large-scale knowledge bases that capture and extend the implicit knowledge of human experts is the foundation for many AI systems. We use an ontology-driven approach for the building, growing and serving of such knowledge bases. This approach relies on several well-known building blocks: document conversion, natural language processing, entity resolution, data transformation and fusion. In this talk, I will discuss wide range of real-world challenges related to the building of these blocks and present our work to address these challenges via better human-machine cooperation.
Meaning Representations for Natural Languages: Design, Models and ApplicationsYunyao Li
EMNLP'2022 Tutorial "Meaning Representations for Natural Languages: Design, Models and Applications"
Instructors: Jeffrey Flanigan, Ishan Jindal, Yunyao Li, Tim O’Gorman, Martha Palmer
Abstract:
We propose a cutting-edge tutorial that reviews the design of common meaning representations, SoTA models for predicting meaning representations, and the applications of meaning representations in a wide range of downstream NLP tasks and real-world applications. Reporting by a diverse team of NLP researchers from academia and industry with extensive experience in designing, building and using meaning representations, our tutorial has three components: (1) an introduction to common meaning representations, including basic concepts and design challenges; (2) a review of SoTA methods on building models for meaning representations; and (3) an overview of applications of meaning representations in downstream NLP tasks and real-world applications. We will also present qualitative comparisons of common meaning representations and a quantitative study on how their differences impact model performance. Finally, we will share best practices in choosing the right meaning representation for downstream tasks.
Natural language understanding is a fundamental task in artificial intelligence. English understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Discovery. However, scaling existing products/services to support additional languages remain an open challenge. In this talk, we will discuss the open challenges in supporting universal natural language understanding. We will share our work in the past few years in addressing these challenges. We will also showcase how universal semantic representation of natural languages can enable cross-lingual information extraction in concrete domains (e.g. compliance) and show ongoing efforts towards seamless scaling existing NLP capabilities across languages with minimal efforts.
Explainability for Natural Language ProcessingYunyao Li
Final deck for our popular tutorial on "Explainability for Natural Language Processing" at KDD'2021. See links below for downloadable version (with higher resolution) and recording of the live tutorial.
Title: Explainability for Natural Language Processing
Presenter: Marina Danilevsky, Shipi Dhanorkar, Yunyao Li and Lucian Popa and Kun Qian and Anbang Xu
Website: http://xainlp.github.io/
Recording: https://www.youtube.com/watch?v=PvKOSYGclPk&t=2s
Downloadable version with higher resolution: https://drive.google.com/file/d/1_gt_cS9nP9rcZOn4dcmxc2CErxrHW9CU/view?usp=sharing
@article{kdd2021xaitutorial,
title={Explainability for Natural Language Processing},
author= {Marina Danilevsky, Shipi Dhanorkar and Yunyao Li and Lucian Popa and Kun Qian and Anbang Xu},
journal={KDD},
year={2021}
}
Abstract:
This lecture-style tutorial, which mixes in an interactive literature browsing component, is intended for the many researchers and practitioners working with text data and on applications of natural language processing (NLP) in data science and knowledge discovery. The focus of the tutorial is on the issues of transparency and interpretability as they relate to building models for text and their applications to knowledge discovery. As black-box models have gained popularity for a broad range of tasks in recent years, both the research and industry communities have begun developing new techniques to render them more transparent and interpretable.Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP/knowledge management researchers, our tutorial has two components: an introduction to explainable AI (XAI) in the NLP domain and a review of the state-of-the-art research; and findings from a qualitative interview study of individuals working on real-world NLP projects as they are applied to various knowledge extraction and discovery at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability inNLP. Then, we will discuss explainability for NLP tasks and reporton a systematic literature review of the state-of-the-art literaturein AI, NLP and HCI conferences. The second component reports on our qualitative interview study, which identifies practical challenges and concerns that arise in real-world development projects that require the modeling and understanding of text data.
Explainability for Natural Language ProcessingYunyao Li
NOTE: Please check out the final version here with small but important updates and links to downloadable version and recording: https://www.slideshare.net/YunyaoLi/explainability-for-natural-language-processing-249992241
Updated version on our popular tutorial on "Explainability for Natural Language Processing" as a tutorial at KDD'2021.
Title: Explainability for Natural Language Processing
@article{kdd2021xaitutorial,
title={Explainability for Natural Language Processing},
author= {Marina Danilevsky, Dhanorkar, Shipi and Li, Yunyao and Lucian Popa and Kun Qian and Anbang Xu},
journal={KDD},
year={2021}
}
Presenter: Marina Danilevsky, Dhanorkar, Shipi and Li, Yunyao and Lucian Popa and Kun Qian and Anbang Xu
Website: http://xainlp.github.io/
Abstract:
This lecture-style tutorial, which mixes in an interactive literature browsing component, is intended for the many researchers and practitioners working with text data and on applications of natural language processing (NLP) in data science and knowledge discovery. The focus of the tutorial is on the issues of transparency and interpretability as they relate to building models for text and their applications to knowledge discovery. As black-box models have gained popularity for a broad range of tasks in recent years, both the research and industry communities have begun developing new techniques to render them more transparent and interpretable.Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP/knowledge management researchers, our tutorial has two components: an introduction to explainable AI (XAI) in the NLP domain and a review of the state-of-the-art research; and findings from a qualitative interview study of individuals working on real-world NLP projects as they are applied to various knowledge extraction and discovery at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability inNLP. Then, we will discuss explainability for NLP tasks and reporton a systematic literature review of the state-of-the-art literaturein AI, NLP and HCI conferences. The second component reports on our qualitative interview study, which identifies practical challenges and concerns that arise in real-world development projects that require the modeling and understanding of text data.
This document brings together a set
of latest data points and publicly
available information relevant for
Business Services Industry. We are
very excited to share this content and
believe that readers will benefit from
this periodic publication immensely.
This document brings together a set of latest data points and publicly available information relevant for Business Services Industry. We are very excited to share this content and believe that readers will benefit immensely from this periodic publication immensely.
Sample Report: Global Online Gaming Market 2015yStats.com
Free Report Samples for our publication "Global Online Gaming Market 2015".
Find the full updated 2019 report available for purchase at: https://ystats.com/shop/global-digital-gaming-market-2019-2/
The “Gartner Perspective: IT Spending” booklet provides an
overview of Gartner research on IT spending and functions as
a reference guide to top-level statistics and IT spending analysis.
It provides a glimpse into the
powerful insight Gartner can
provide as you navigate through
what may be the most important
year of your career.
Rong Viet Securities - Investment Strategy June 2018Thomas Farthofer
In their recently published strategy report for June 2018, our partner Rong Viet explains why, despite a severe correction in the stock market during April and May, investors are in no need to rush in to buy massively. Still, valuations now appear more reasonable and it seems that it is time to gradually accumulate stocks.
Access to this presentation has been made possible through "Sao Bien. Room for Education", an Austrian-based non-profit organization and cooperation partner of Viet Dragon Securities.
Reprinted with the permission of Viet Dragon Securities. Not for US investors.
Sample Report: Europe Online Gaming Market 2015yStats.com
Free Report Samples for our publication "Europe Online Gaming Market 2015".
Find the full updated "Global Digital Gaming Market 2019" report available for purchase at: https://ystats.com/shop/global-digital-gaming-market-2019-2/
This document brings together a set of latest data points and publicly available information relevant for Agile & AI Operations. We are very excited to share this content and believe that readers will benefit immensely from this periodic publication immensely.
The Role of Patterns in the Era of Large Language ModelsYunyao Li
Slides for my keynote at PAN-DL Workshop (Pattern-based Approaches to NLP in the Age of Deep Learning) at EMNLP'2023 (December. 6, 2023).
In this talk, I share our initial learnings from constructing, growing and serving large knowledge graphs
Building, Growing and Serving Large Knowledge Graphs with Human-in-the-LoopYunyao Li
Keynote talk at HILDA'2023 at SIGMOD on June 18, 2023.
Abstract: The ability to build large-scale knowledge bases that capture and extend the implicit knowledge of human experts is the foundation for many AI systems. We use an ontology-driven approach for the building, growing and serving of such knowledge bases. This approach relies on several well-known building blocks: document conversion, natural language processing, entity resolution, data transformation and fusion. In this talk, I will discuss wide range of real-world challenges related to the building of these blocks and present our work to address these challenges via better human-machine cooperation.
Meaning Representations for Natural Languages: Design, Models and ApplicationsYunyao Li
EMNLP'2022 Tutorial "Meaning Representations for Natural Languages: Design, Models and Applications"
Instructors: Jeffrey Flanigan, Ishan Jindal, Yunyao Li, Tim O’Gorman, Martha Palmer
Abstract:
We propose a cutting-edge tutorial that reviews the design of common meaning representations, SoTA models for predicting meaning representations, and the applications of meaning representations in a wide range of downstream NLP tasks and real-world applications. Reporting by a diverse team of NLP researchers from academia and industry with extensive experience in designing, building and using meaning representations, our tutorial has three components: (1) an introduction to common meaning representations, including basic concepts and design challenges; (2) a review of SoTA methods on building models for meaning representations; and (3) an overview of applications of meaning representations in downstream NLP tasks and real-world applications. We will also present qualitative comparisons of common meaning representations and a quantitative study on how their differences impact model performance. Finally, we will share best practices in choosing the right meaning representation for downstream tasks.
Natural language understanding is a fundamental task in artificial intelligence. English understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Discovery. However, scaling existing products/services to support additional languages remain an open challenge. In this talk, we will discuss the open challenges in supporting universal natural language understanding. We will share our work in the past few years in addressing these challenges. We will also showcase how universal semantic representation of natural languages can enable cross-lingual information extraction in concrete domains (e.g. compliance) and show ongoing efforts towards seamless scaling existing NLP capabilities across languages with minimal efforts.
Explainability for Natural Language ProcessingYunyao Li
Final deck for our popular tutorial on "Explainability for Natural Language Processing" at KDD'2021. See links below for downloadable version (with higher resolution) and recording of the live tutorial.
Title: Explainability for Natural Language Processing
Presenter: Marina Danilevsky, Shipi Dhanorkar, Yunyao Li and Lucian Popa and Kun Qian and Anbang Xu
Website: http://xainlp.github.io/
Recording: https://www.youtube.com/watch?v=PvKOSYGclPk&t=2s
Downloadable version with higher resolution: https://drive.google.com/file/d/1_gt_cS9nP9rcZOn4dcmxc2CErxrHW9CU/view?usp=sharing
@article{kdd2021xaitutorial,
title={Explainability for Natural Language Processing},
author= {Marina Danilevsky, Shipi Dhanorkar and Yunyao Li and Lucian Popa and Kun Qian and Anbang Xu},
journal={KDD},
year={2021}
}
Abstract:
This lecture-style tutorial, which mixes in an interactive literature browsing component, is intended for the many researchers and practitioners working with text data and on applications of natural language processing (NLP) in data science and knowledge discovery. The focus of the tutorial is on the issues of transparency and interpretability as they relate to building models for text and their applications to knowledge discovery. As black-box models have gained popularity for a broad range of tasks in recent years, both the research and industry communities have begun developing new techniques to render them more transparent and interpretable.Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP/knowledge management researchers, our tutorial has two components: an introduction to explainable AI (XAI) in the NLP domain and a review of the state-of-the-art research; and findings from a qualitative interview study of individuals working on real-world NLP projects as they are applied to various knowledge extraction and discovery at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability inNLP. Then, we will discuss explainability for NLP tasks and reporton a systematic literature review of the state-of-the-art literaturein AI, NLP and HCI conferences. The second component reports on our qualitative interview study, which identifies practical challenges and concerns that arise in real-world development projects that require the modeling and understanding of text data.
Explainability for Natural Language ProcessingYunyao Li
NOTE: Please check out the final version here with small but important updates and links to downloadable version and recording: https://www.slideshare.net/YunyaoLi/explainability-for-natural-language-processing-249992241
Updated version on our popular tutorial on "Explainability for Natural Language Processing" as a tutorial at KDD'2021.
Title: Explainability for Natural Language Processing
@article{kdd2021xaitutorial,
title={Explainability for Natural Language Processing},
author= {Marina Danilevsky, Dhanorkar, Shipi and Li, Yunyao and Lucian Popa and Kun Qian and Anbang Xu},
journal={KDD},
year={2021}
}
Presenter: Marina Danilevsky, Dhanorkar, Shipi and Li, Yunyao and Lucian Popa and Kun Qian and Anbang Xu
Website: http://xainlp.github.io/
Abstract:
This lecture-style tutorial, which mixes in an interactive literature browsing component, is intended for the many researchers and practitioners working with text data and on applications of natural language processing (NLP) in data science and knowledge discovery. The focus of the tutorial is on the issues of transparency and interpretability as they relate to building models for text and their applications to knowledge discovery. As black-box models have gained popularity for a broad range of tasks in recent years, both the research and industry communities have begun developing new techniques to render them more transparent and interpretable.Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP/knowledge management researchers, our tutorial has two components: an introduction to explainable AI (XAI) in the NLP domain and a review of the state-of-the-art research; and findings from a qualitative interview study of individuals working on real-world NLP projects as they are applied to various knowledge extraction and discovery at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability inNLP. Then, we will discuss explainability for NLP tasks and reporton a systematic literature review of the state-of-the-art literaturein AI, NLP and HCI conferences. The second component reports on our qualitative interview study, which identifies practical challenges and concerns that arise in real-world development projects that require the modeling and understanding of text data.
Human in the Loop AI for Building Knowledge Bases Yunyao Li
The ability to build large-scale domain-specific knowledge bases that capture and extend the implicit knowledge of human experts is the foundation for many AI systems. We use an ontology-driven approach for the creation, representation and consumption of such domain-specific knowledge bases. This approach relies on several well-known building blocks: natural language processing, entity resolution, data transformation and fusion. I will present several human-in-the-loop work that target domain experts (rather than programmers) to extract the domain knowledge from the human expert and map it into the "right" models or algorithms. I will also share successful use cases in several domains, including Compliance, Finance, and Healthcare: by using these tools we can match the level of accuracy achieved by manual efforts, but at a significantly lower cost and much higher scale and automation.
Slides for talk given at Women in Engineering on March 20, 2021.
Abstract:
Natural language understanding is a fundamental task in artificial intelligence. English understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Discovery. However, scaling existing products/services to support additional languages remain an open challenge. In this talk, we will discuss the open challenges in supporting universal natural language understanding. We will share our work in the past few years in addressing these challenges. We will also showcase how universal semantic representation of natural languages can enable cross-lingual information extraction in concrete domains (e.g. compliance) and show ongoing efforts towards seamless scaling existing NLP capabilities across languages with minimal efforts.
Explainability for Natural Language ProcessingYunyao Li
Tutorial at AACL'2020 (http://www.aacl2020.org/program/tutorials/#t4-explainability-for-natural-language-processing).
More recent version: https://www.slideshare.net/YunyaoLi/explainability-for-natural-language-processing-249912819
Title: Explainability for Natural Language Processing
@article{aacl2020xaitutorial,
title={Explainability for Natural Language Processing},
author= {Dhanorkar, Shipi and Li, Yunyao and Popa, Lucian and Qian, Kun and Wolf, Christine T and Xu, Anbang},
journal={AACL-IJCNLP 2020},
year={2020}
Presenter: Shipi Dhanorkar, Christine Wolf, Kun Qian, Anbang Xu, Lucian Popa and Yunyao Li
Video: https://www.youtube.com/watch?v=3tnrGe_JA0s&feature=youtu.be
Abstract:
We propose a cutting-edge tutorial that investigates the issues of transparency and interpretability as they relate to NLP. Both the research community and industry have been developing new techniques to render black-box NLP models more transparent and interpretable. Reporting from an interdisciplinary team of social science, human-computer interaction (HCI), and NLP researchers, our tutorial has two components: an introduction to explainable AI (XAI) and a review of the state-of-the-art for explainability research in NLP; and findings from a qualitative interview study of individuals working on real-world NLP projects at a large, multinational technology and consulting corporation. The first component will introduce core concepts related to explainability in NLP. Then, we will discuss explainability for NLP tasks and report on a systematic literature review of the state-of-the-art literature in AI, NLP, and HCI conferences. The second component reports on our qualitative interview study which identifies practical challenges and concerns that arise in real-world development projects which include NLP.
Towards Universal Language Understanding (2020 version)Yunyao Li
Keynote talk given at Pacific Asia Conference on Language, Information and Computation (PACLIC 34) on Pacific Asia Conference on Language, Information and Computation (PACLIC 34) on October 24, 2020.
Title: Towards Universal Natural Language Understanding
Abstract:
Understanding the semantics of the natural language is a fundamental task in artificial intelligence. English semantic understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Compare and Comply. However, scaling existing products/services to support additional languages remain an open challenge. In this talk, we will discuss the open challenges in supporting universal natural language understanding. We will share our work in addressing these challenges in the past few years to provide the same unified semantic representation across languages. We will also showcase how such universal semantic understanding of natural languages can enable cross-lingual information extraction in concrete domains (e.g. insurance and compliance) and show promise towards seamless scaling existing NLP capabilities across languages with minimal efforts.
Towards Universal Semantic Understanding of Natural LanguagesYunyao Li
Keynote talk at TextXD 2019(https://www.textxd.org)
Abstract:
Understanding the semantics of the natural language is a fundamental task in artificial intelligence. English semantic understanding has reached a mature state and successfully deployed in multiple IBM AI products and services, such as Watson Natural Language Understanding and Watson Compare and Comply. However, scaling existing products/services to support additional languages remain an open challenge. In this demo, we will present Polyglot, a multilingual semantic parser capable of semantically parsing sentences in 9 different languages from 4 different language groups into the same unified semantic representation. We will also showcase how such universal semantic understanding of natural languages can enable cross-lingual information extraction in concrete domains (e.g. insurance and compliance) and show promise towards seamless scaling existing NLP capabilities across languages with minimal efforts.
An In-depth Analysis of the Effect of Text Normalization in Social MediaYunyao Li
Poster corresponding to our NAACL'2015 paper "An In-depth Analysis of the Effect of Text Normalization in Social Media"
Abstract: Recent years have seen increased interest in text normalization in social media, as the in-formal writing styles found in Twitter and other social media data often cause problems for NLP applications. Unfortunately, most current approaches narrowly regard the nor- malization task as a “one size fits all" task of replacing non-standard words with their standard counterparts. In this work we build a taxonomy of normalization edits and present a study of normalization to examine its effect on three different downstream applications (de- pendency parsing, named entity recognition, and text-to-speech synthesis). The results sug- gest that how the normalization task should be viewed is highly dependent on the targeted application. The results also show that normalization must be thought of as more than word replacement in order to produce results comparable to those seen on clean text.
Paper: https://www.aclweb.org/anthology/N15-1045
Exploiting Structure in Representation of Named Entities using Active LearningYunyao Li
Slides for our COLING'18 paper: http://aclweb.org/anthology/C18-1058
Fundamental to several knowledge-centric applications is the need to identify named entities from their textual mentions. However, entities lack a unique representation and their mentions can differ greatly. These variations arise in complex ways that cannot be captured using textual similarity metrics. However, entities have underlying structures, typically shared by entities of the same entity type, that can help reason over their name variations. Discovering, learning and manipulating these structures typically requires high manual effort in the form of large amounts of labeled training data and handwritten transformation programs. In this work, we propose an active-learning based framework that drastically reduces the labeled data required to learn the structures of entities. We show that programs for mapping entity mentions to their structures can be automatically generated using human-comprehensible labels. Our experiments show that our framework consistently outperforms both handwritten programs and supervised learning models. We also demonstrate the utility of our framework in relation extraction and entity resolution tasks.
K-SRL: Instance-based Learning for Semantic Role LabelingYunyao Li
Slides for our COLING'16 paper http://aclweb.org/anthology/C/C16/C16-1058.pdf
Abstract:
Semantic role labeling (SRL) is the task of identifying and labeling predicate-argument structures in sentences with semantic frame and role labels. A known challenge in SRL is the large number of low-frequency exceptions in training data, which are highly context-specific and difficult to generalize. To overcome this challenge, we propose the use of instance-based learning that performs no explicit generalization, but rather extrapolates predictions from the most similar instances in the training data. We present a variant of k-nearest neighbors (kNN) classification with composite features to identify nearest neighbors for SRL. We show that high-quality predictions can be derived from a very small number of similar instances. In a comparative evaluation we experimentally demonstrate that our instance-based learning approach significantly outperforms current state-of-the-art systems on both in-domain and out-of-domain data, reaching F1-scores
of 89,28% and 79.91% respectively
Natural Language Data Management and Interfaces: Recent Development and Open ...Yunyao Li
Slides deck for SIGMOD 2017 Tutorial.
ABSTRACT:
The volume of natural language text data has been rapidly increasing over the past two decades, due to factors such as the growth of the Web, the low cost associated to publishing and the progress on the digitization of printed texts. This growth combined with the proliferation of natural language systems for search and retrieving information provides tremendous opportunities for studying some of the areas where database systems and natural language processing systems overlap. This tutorial explores two more relevant areas of overlap to the database community: (1) managing
natural language text data in a relational database, and (2) developing natural language interfaces to databases. The tutorial presents state-of-the-art methods, related systems, research opportunities and challenges covering both area.
Polyglot: Multilingual Semantic Role Labeling with Unified LabelsYunyao Li
Poster for our ACL paper "Polyglot: Multilingual Semantic Role Labeling with Unified Labels".
Abstract:
We present POLYGLOT, a semantic role labeling system capable of semantically parsing sentences in 9 different languages from 4 different language groups. A core differentiator is that this system predicts English Proposition Bank labels for all supported languages. This means that
for instance a Japanese sentence will be tagged with the same labels as an English sentence with similar semantics would be. This is made possible by training the system with target language data that was automatically labeled with English PropBank labels using an annotation projection approach. We give an overview of our system, the automatically produced training data, and discuss possible applications
and limitations of this work. We present a demonstrator that accepts sentences in English, German, French, Spanish, Japanese, Chinese, Arabic, Russian and Hindi and
outputs a visualization of its shallow semantics.
Invited Talk at Modern Data Management Systems Summit on August 29-30, 2014 at Tsinghua University in Beijing, China.
http://ise.thss.tsinghua.edu.cn/MDMS/English/program.jsp
Abstract:
Modern enterprises are increasingly relying on complex analyses on large data sets to drive business decisions. Tasks such as root cause analysis from system logs and lead generation based on social media, customer retention and digital marketing are rapidly gaining importance. These applications generally consist of three major analytic phases: text analytics, semi-structured data processing (joins, group-by, aggregation), and statistical/predictive modeling. The size of the datasets in conjunction with the complexity of the analysis necessitates large-scale distributed processing of the analytical algorithms. At IBM we are building tools and technologies based on declarative languages to support each of these analytic phases. The declarative nature of the language abstracts away the need for programmer-optimization. Furthermore, the syntax of these languages is designed to appeal to the corresponding communities. As an example for statistical modeling, we expose a high-level language with syntax similar to R -- a very popular statistical processing language.
In this talk I will give an overview of some real-world big data applications we are currently working on and use that to motivate the need for declarative analytics consisting of the three major phases discussed above. I will then describe, in some detail, declarative systems for text analytics along with a discussion on speeds, feeds and comparisons.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Elevating Tactical DDD Patterns Through Object Calisthenics
Towards Deep Table Understanding
1. Towards Deep Table
Understanding
Yunyao Li
Distinguished Research Staff Member
Senior Manager
Scalable Knowledge Intelligence Department
IBM Research – Almaden
Twitter: @yunyao_li
Email: yunyaoli@us.ibm.com
DIS @KDD’2021
2. What is document conversion?
2
Target: Content in an open text-based
format
(JSON, HTML, etc.)
Amenable for NLP/AI driven understanding of its
contents
IBM Confidential
Source: Business documents
(PDF/Word/Image)
Designed for document exchange, printing,
human consumption and archival
3. What is Document Conversion ?
An informal view
IBM Confidential 3
Preserving the reading
order
Identify the correct sequencing
of text, tabular and image
content mimicking human
reading order
Extracting internal structure
of complex document
elements
For example, for tables, extracting
table cells, column and row headers,
individual cell values, table title, etc.
Identifying and extracting
document structure
For example, extracting document
elements such as sections and
paragraphs, tables, and
relationships among them.
4. Why is it Hard?
Noise
artifact
s
Fuzzy
Text &
Skew
Colored
backgrou
nd
Scanned documents
Table with
visual clues
only
Multi-row, multi-
column column
headers
Nested
row
headers
Tables with
Textual content
Table with
graphic
lines
Variety of tables Various elements
Line Charts
Histogram
Pie Charts
Signatures
Logos
5. Deep Document Understanding
Understanding of complex documents
– Visual elements such as charts, tables, photos, images,
sketches, etc.
– PDF manuals and technical guides, financial and scientific
documents, scanned invoices and passports, etc.
Focus on converting challenging documents
– in terms of structure, content and quality
Source: If applicable, describe source origin
5
https://www.research.ibm.com/blog/deep-document-understanding-complex-documents
6. Why is it Hard?
Noise
artifact
s
Fuzzy
Text &
Skew
Colored
backgrou
nd
Scanned documents
Table with
visual clues
only
Multi-row, multi-
column column
headers
Nested
row
headers
Tables with
Textual content
Table with
graphic
lines
Variety of tables Various elements
Line Charts
Histogram
Pie Charts
Signatures
Logos
7. Input: Document contents in native format
- PDF
- Image
- Office Docs
- …
Table Extraction: Problem Definition
Output: Document contents with tabular information:
1) Table border for each table
2) Partitioning table contents into cells
3) Both vertical and horizontal alignment of cells
8. Tables and Cells as Objects Leverage object detection improvements with deep
learning networks to detect tables and cells
80
During fiscal 2016, the Company acquired a direct interest in Vice for $400 million of cash, and at September 29, 2018
owned an 11% interest. The Company accounts for its interest in Vice as an equity method investment.
During fiscal 2018, the Company recorded a $157 million impairment of its interest in Vice.
Hulu
At the end of fiscal 2015, the Company had a 33% interest in Hulu, a joint venture owned one-third each by the
Company, 21CF and Comcast Corporation. Warner Media LLC (WM) acquired a 10% interest from Hulu for $0.6 billion in
August 2016, which diluted the Company’s ownership interest to 30%. In addition, WM has made $0.2 billion in subsequent
capital contributions. For not more than 36 months from August 2016, WM has the right to sell its shares to Hulu and Hulu has
the right to purchase the shares from WM under certain limited circumstances arising from regulatory review. The Company
and 21CF have agreed to make a capital contribution for up to approximately $0.4 billion each if Hulu is required to repurchase
WM’s shares. The August 2016 transaction resulted in a deemed sale by the Company of a portion of its interest in Hulu at a
gain of approximately $175 million. The Company expects to recognize the gain if and when the put and call options expire.
Following completion of the 21CF acquisition the Company will consolidate Hulu’ s financial results and assume 21CF’s
capital contribution obligations.
The Company accounts for its interest in Hulu as an equity method investment.
Goodwill
The changes in the carrying amount of goodwill for the years ended September 29, 2018 and September 30, 2017 are as
follows:
Media
Networks
Parks and
Resorts
Studio
Entertainment
Consumer
Products &
Interactive
Media Unallocated (1)
Total
Balance at Oct. 1, 2016 $ 16,345 $ 291 $ 6,830 $ 4,344 $ — $ 27,810
Acquisitions — — — — 3,600 3,600
Dispositions — — — — — —
Other, net (20) — (13) 49 — 16
Balance at Sept. 30, 2017 $ 16,325 $ 291 $ 6,817 $ 4,393 $ 3,600 $ 31,426
Acquisitions — — — — — —
Dispositions — — — — — —
Other, net 3,063 — 347 33 (3,600) (157)
Balance at Sept. 29, 2018 $ 19,388 $ 291 $ 7,164 $ 4,426 $ — $ 31,269
(1)
Other, net primarily represents the allocation of BAMTech goodwill to segments based on the final purchase price
allocation and also includes the impact of updates to our initial estimated fair value of intangible assets related to
BAMTech.
4 Other Income, net
Other income, net is as follows:
2018 2017 2016
Gains on sales of real estate and property rights $ 560 $ — $ —
Settlement of litigation 38 (177) —
Gain related to the acquisition of BAMTech 3 255 —
Other income, net $ 601 $ 78 $ —
Gains from sales of real estate and property rights
In fiscal 2018, the Company recorded gains of $560 million in connection with the sale of real estate and property rights
in New York City.
Settlement of litigation
In fiscal 2018, the Company recorded $38 million in insurance recoveries in connection with the settlement of a litigation
matter for which the Company recorded a charge of $177 million, net of committed insurance recoveries in fiscal 2017.
9. But where do we get the data???
Data
ICDAR 2013 Dataset [1]: 256 table examples with table structure
ICDAR 2019 Dataset [2]: 80 page images with table structure
Marmot dataset [3]: 2000 total labelled PDF pages with 1349 tables
TableBank [4]: 145k labelled table examples … but labelled tables
structures only have logical coordinates
[1] M. Göbel et al., ICDAR 2013 Table Competition ICDAR 2013
[2] L.Gao et. al., ICDAR 2019 Competition on table detection and recognition (CTDAR). ICDAR 2019
[3] http://www.icst.pku.edu.cn/cpdp/data/marmot_data.htm
[4] Minghao Li et al. Tablebank: Table benchmark for image-based table detection and recognition. LREC 2020
Too small for training
Missing critical information
10. PubTabNet Dataset
Enhanced scientific dataset with cell bounding boxes with automated text matching.
568K Tables from PubMed
ibm.biz/pubtabnet
Zhong et al. Image-Based Table Recognition: Data, Model, and Evaluation ECCV 2020
11. FinTabNet Dataset
New dataset with auto-generated table boundary, cell boundary and structure annotations.
>110k Tables from annual reports of S&P 500 companies.
ibm.biz/fintabnet
13. Motivation for a Specialized Table
and Cell Detector
Tables and cells have very different sizes and aspect ratios,
unlike the classic object detection classes
Cells are always inside tables and tables always contain cells
80
During fiscal 2016, the Company acquired a direct interest in Vice for $400 million of cash, and at September 29, 2018
owned an 11% interest. The Company accounts for its interest in Vice as an equity method investment.
During fiscal 2018, the Company recorded a $157 million impairment of its interest in Vice.
Hulu
At the end of fiscal 2015, the Company had a 33% interest in Hulu, a joint venture owned one-third each by the
Company, 21CF and Comcast Corporation. Warner Media LLC (WM) acquired a 10% interest from Hulu for $0.6 billion in
August 2016, which diluted the Company’s ownership interest to 30%. In addition, WM has made $0.2 billion in subsequent
capital contributions. For not more than 36 months from August 2016, WM has the right to sell its shares to Hulu and Hulu has
the right to purchase the shares from WM under certain limited circumstances arising from regulatory review. The Company
and 21CF have agreed to make a capital contribution for up to approximately $0.4 billion each if Hulu is required to repurchase
WM’s shares. The August 2016 transaction resulted in a deemed sale by the Company of a portion of its interest in Hulu at a
gain of approximately $175 million. The Company expects to recognize the gain if and when the put and call options expire.
Following completion of the 21CF acquisition the Company will consolidate Hulu’ s financial results and assume 21CF’s
capital contribution obligations.
The Company accounts for its interest in Hulu as an equity method investment.
Goodwill
The changes in the carrying amount of goodwill for the years ended September 29, 2018 and September 30, 2017 are as
follows:
Media
Networks
Parks and
Resorts
Studio
Entertainment
Consumer
Products &
Interactive
Media Unallocated (1)
Total
Balance at Oct. 1, 2016 $ 16,345 $ 291 $ 6,830 $ 4,344 $ — $ 27,810
Acquisitions — — — — 3,600 3,600
Dispositions — — — — — —
Other, net (20) — (13) 49 — 16
Balance at Sept. 30, 2017 $ 16,325 $ 291 $ 6,817 $ 4,393 $ 3,600 $ 31,426
Acquisitions — — — — — —
Dispositions — — — — — —
Other, net 3,063 — 347 33 (3,600) (157)
Balance at Sept. 29, 2018 $ 19,388 $ 291 $ 7,164 $ 4,426 $ — $ 31,269
(1)
Other, net primarily represents the allocation of BAMTech goodwill to segments based on the final purchase price
allocation and also includes the impact of updates to our initial estimated fair value of intangible assets related to
BAMTech.
4 Other Income, net
Other income, net is as follows:
2018 2017 2016
Gains on sales of real estate and property rights $ 560 $ — $ —
Settlement of litigation 38 (177) —
Gain related to the acquisition of BAMTech 3 255 —
Other income, net $ 601 $ 78 $ —
Gains from sales of real estate and property rights
In fiscal 2018, the Company recorded gains of $560 million in connection with the sale of real estate and property rights
in New York City.
Settlement of litigation
In fiscal 2018, the Company recorded $38 million in insurance recoveries in connection with the settlement of a litigation
matter for which the Company recorded a charge of $177 million, net of committed insurance recoveries in fiscal 2017.
14. Global Table Extractor (GTE)
Zheng et al. A Deep Learning Framework for Joint Table Identification and Cell Structure Recognition Using Visual Context. WACV. 2021
15. Penalty Loss:
A loss on the table classification
where the penalty is applied if:
1. The detection is a table but contains very little
cell mask inside the bounding box.
2. The detection is not a table but contains a lot
cells inside the bounding box.
Penalty Loss
16. When tables with similar confidence are overlapping,
we select the table based on its “table-ness”.
1. Have lots of cellular regions.
2. NOT have cellular regions just outside that is not
being covered by a non-overlapping region.
Cell-Aware Table Reranking
17. SoTA on Table Border Detection
ICDAR 2013
Character level recall,
precision and F1
ICDAR 2019
Precision and Recall at different
levels of Intersection Over Union
(IOU)
18. Tables Come in Different Styles Object detection models focus on local areas and
cannot handle nuances of different global table styles.
Table 8.12 - Own brand shares (food only) for leading retailers, 1996
Names Own brands market share Number of items
Franprix 28.0 n.a.
Casino 24.8 1800
Intermarché 24.7 2500
Géant 20.0 1800
Carrefour 18.9 1642
Monoprix 18.7 1800
Système U 18.5 985
Continent 17.8 1440
Stoc 16.2 650
Auchan 15.7 1500
Match 15.4 1100
Champion 15.1 1240
Leclerc 14.8 500
Cora 12.2 1224
Prisunic 11.7 550
Source: Secodip-linéaires, 1997
Table 8.13 illustrates the development of own brand market shares in supermarkets and hypermarkets:
Table 8.13 - National brands, Own brand and low price items shares for supermarkets and
hypermarkets
1991 1994 1995 1996
National Brands 80.6 75.0 75.3 76.0
Own Brands 14.7 17.1 17.4 17.1
Low price items 4.7 7.9 7.3 6.9
Source: LSA, 1998
Own brands growth is a major trend in the recent evolution of distribution. In 1995, own brands provided
on average 20% of sales (25% of shelf space) compared to only 10% ten years before and the development
goes on. Leclerc, for instance, which was initially opposed to own brand development (less than 7% of
sales before 1997 were own brand), changed its strategy in 1997 and its goal is now to double its own
brands turnover. Retailers try to reinforce the association between their name and their own products.
Table of Contents
Canada Dollar Tree
Alberta 37
British Columbia 49
Manitoba 13
Ontario 110
Saskatchewan 16
Total 225
We lease the vast majority of our stores and expect to lease the majority of our new stores as we expand. Our leases typically
provide for a short initial lease term, generally five years, with options to extend; however, in some cases we have initial lease
terms of seven to fifteen years. We believe this leasing strategy enhances our flexibility to pursue various expansion opportunities
resulting from changing market conditions. As current leases expire, we believe that we will be able to obtain lease renewals, if
desired, for present store locations, or to obtain leases for equivalent or better locations in the same general area.
Distribution Centers
The following table includes information about the distribution centers that we operate in the United States. Except for 0.4
million square feet of our distribution center in San Bernardino, California, all of our distribution center capacity is owned. In
2018, we completed our Warrensburg, Missouri distribution center, which is 1.2 million square feet, automated and currently serves
stores in our Dollar Tree segment. In 2016, we completed our 1.5 million square foot Cherokee County, South Carolina distribution
center and expanded our Stockton, California distribution center by 0.3 million square feet. Our St. George, Utah distribution
center services both Family Dollar and Dollar Tree stores. In addition, we ship select product from our Dollar Tree distribution
centerstoourFamilyDollardistributioncentersandinfiscal2019,weexpecttoship selectproductfromourDollarTree distribution
centers directly to certain of our Family Dollar stores. We believe our distribution center network is currently capable of supporting
approximately $28.0 billion in annual sales in the United States.
Dollar Tree Square Family Dollar Square
Distribution Centers Footage Distribution Centers Footage
Chesapeake, Virginia 400,000 Matthews, North Carolina 930,000
Olive Branch, Mississippi 425,000 West Memphis, Arkansas 850,000
Joliet, Illinois 1,470,000 Front Royal, Virginia 907,000
Stockton, California 854,000 Duncan, Oklahoma 907,000
Savannah, Georgia 1,014,000 Morehead, Kentucky 907,000
Briar Creek, Pennsylvania 1,003,000 Maquoketa, Iowa 907,000
Marietta, Oklahoma 1,004,000 Odessa, Texas 907,000
San Bernardino, California 802,000 Marianna, Florida 907,000
Ridgefield, Washington 665,000 Rome, New York 907,000
Windsor, Connecticut 1,001,000 Ashley, Indiana 814,000
Cherokee County, South Carolina 1,512,000 St. George, Utah* 814,000
Warrensburg, Missouri 1,200,000
*Services both Dollar Tree and Family Dollar stores
In 2018, we began construction on our Morrow County, Ohio distribution center, which will be 1.2 million square feet and
automated, and will initially serve stores in our Dollar Tree segment. We expect this facility to be operational in the third quarter
of 2019. In fiscal 2019, we announced plans to construct a new 1.2 million square foot distribution center in Rosenberg, Texas
which is expected to provide service directly to Dollar Tree and Family Dollar stores and be operational by the summer of 2020.
All future distribution centers will open with the capability to service both Dollar Tree and Family Dollar stores.
Each of our distribution centers contains advanced materials handling technologies, including radio-frequency inventory
tracking equipment and specialized information systems. With the exception of our Ridgefield, Washington facility and our
Matthews, North Carolina facility, each of our distribution centers in the United States also contains automated conveyor and
sorting systems.
Distribution services in Canada are provided by a third party from facilities in British Columbia and Ontario.
21
42 Styrene-Acrylonitrile Trimer, NTP TR 573
Postnatal Day 1 Postnatal Day 4 Postnatal Day 7 Postnatal Day 14 Postnatal Day 20
Concentration
(ppm)
Body
Weight
(g)
Weight
Relative
to
Controls
(%)
Body
Weight
(g)
Weight
Relative
to
Controls
(%)
Body
Weight
(g)
Weight
Relative
to
Controls
(%)
Body
Weight
(g)
Weight
Relative
to
Controls
(%)
Body
Weight
(g)
Weight
Relative
to
Controls
(%)
Male
0 39 5.8 10 8.8 13.5 25.7 35.0
250 30 5.9 102 10 9.0 102 13.6 101 26.2 102 35.8 102
500 33 6.0 103 10 8.6 98 13.0 96 25.1 98 34.7 99
1,000 31 5.8 100 10 8.4 96 13.0 96 24.9 97 34.5 99
2,000 38 5.8 100 10 8.8 100 12.9 96 24.6 96 31.6** 90
4,000 27 5.3** 91 10 7.5** 85 10.4** 77 16.8** 65 19.8** 57
Female
0 23 5.4 10 8.2 12.7 24.7 33.6
250 34 5.6 104 10 8.5 104 12.9 102 24.7 100 33.9 101
500 32 5.4 100 10 8.2 100 12.7 100 25.0 101 33.6 100
1,000 40 5.5 102 10 8.0 98 12.4 98 24.2 98 32.7 97
2,000 49 5.3 98 10 8.2 100 12.4 98 23.7 96 30.3** 90
4,000 31 5.0** 93 10 7.3* 89 9.9** 78 16.1** 65 18.8** 56
* Significantly different (P≤0.05) from the control group by Dunnett’s test
** P≤0.01
a
Weights are given as group means.
b Number of animals weighed on postnatal day 1
c Number of animals weighed on postnatal days 4, 7, 14, and 20
No. No.
TABLE 6
Mean Body Weights of F1 Pups to Postnatal Day 20
in the 7-Week Perinatal and Postnatal Feed Study of Styrene-Acrylonitrile Trimer
20. Cell Boundary to Structure with
Position Clustering
1. Sample at each box center horizontally and
vertically to determine the number of rows and
columns in table
2. Determine alignment for columns and rows
based on cell bounding box distance
3. Use K-means clustering with number of
centers from step 3 for each direction with
coordinates depending on alignment
4.Assign cell boxes to each row/column
depending on alignment
25. Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLUE 3.40 3.30
ALK 3.70 3.40
Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLUE 3.40 3.30
ALK 3.70 3.40
Original Table: Ground Truth:
Let’s Look an Example
26. Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLU
E
3.40 3.30
ALK 3.70 3.40
Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLUE 3.40 3.30
ALK 3.70 3.40
Candidate Table A Candidate Table B
Which One is Better?
27. Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLU
E
3.40 3.30
ALK 3.70 3.40
Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLUE 3.40 3.30
ALK 3.70 3.40
Candidate Table A
Candidate Table B
Let’s Compute the Metrics ... Wait … Which One?
ICDAR 2013 Candidate Table A Candidate Table B
P R F1 P R F1
Table Area 1 0.79 0.88 1 0.81 0.90
ICDAR 2019 Candidate Table A Candidate Table B
IOU IOU
Table Area 0.8 0.8
28. Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLU
E
3.40 3.30
ALK 3.70 3.40
Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLUE 3.40 3.30
ALK 3.70 3.40
Candidate Table A
Candidate Table B
Let’s Compute the Metrics ... Wait … Which One?
ICDAR 2013 Candidate Table A Candidate Table B
P R F1 P R F1
Table Area 1 0.79 0.88 1 0.81 0.90
Cell Adjacency 1 0.76 0.86 1 0.88 0.94
ICDAR 2019 Candidate Table A Candidate Table B
IOU IOU
Table Area 0.8 0.8
29. Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLU
E
3.40 3.30
ALK 3.70 3.40
Revenue ($Bn)
2008 2009
AAL 23.8 19.9
LUV 11.0 10.4
SAVE 1.10 0.70
ULCC 1.40 1.10
UAL 20.2 16.3
DAL 22.7 28.1
JBLUE 3.40 3.30
ALK 3.70 3.40
Candidate Table A
Candidate Table B
Let’s Compute the Metrics ... Wait … Which One?
ICDAR 2013 Candidate Table A Candidate Table B
P R F1 P R F1
Table Area 1 0.79 0.88 1 0.81 0.90
Cell Adjacency 1 0.76 0.86 1 0.88 0.94
ICDAR 2019 Candidate Table A Candidate Table B
IOU IOU
Table Area 0.8 0.8
Downstream
Applications
Candidate Table A Candidate Table B
P R F1 P R F1
Functional metric[1] 1 0.75 0.86 1 0 0
[1] Max C. Göbel, Tamir Hassan, Ermelinda Oro, Giorgio Orsi: A
methodology for evaluating algorithms for table understanding in PDF
documents. Document Engineering 2012
Keep information to
recover semantics
for most data cells
Missing column
headers Cannot
recover semantics
for the data cells
31. How to make it works better?
Retraining with interactive labeling
32. TableLab: Interactive Labeling
32
Nancy Xin Ru Wang, Douglas Burdick, Yunyao Li: TableLab: An Interactive Table Extraction System with Adaptive Deep Learning. IUI Companion 2021
33. Preliminary Experimental Results for Retraining
Method CEDAR EDGAR Invoices Appraisals Health Docs
GTE 0.94 0.84 0.47 0.85 0.93
GTE with
Retraining
0.96 0.91 0.92 0.96 0.98
Dataset
20 pages with tables per category:
10 for retraining, 10 for testing
Cell Adjacency Detection
Method CEDAR EDGAR Invoices Appraisals Health Docs
GTE 0.88 0.62 0.42 0.71 0.55
GTE with
Retraining
0.90 0.82 0.68 0.90 0.77
Evaluation Metric
F1 metric for Table Boundary and
Cell Adjacency as defined in [1]
[1] Göbel et al. “A Methodology for Evaluating
Algorithms for Table Understanding in PDF
Documents”. DocEng '12
Table Boundary Detection
Summary: Retraining is effective even with small amount of labeled data
34. Example Use Cases – Research & Social Good
34
CORD-19 [Wang et al, ACL-CORD-19’21]
Weather.com COVID-19 Dashboard
Table QA [Fauceglia et al, AAAI’21, Glass et al, NAACL’21]
Better Understand Climate Change via Historical Records
35. Example Use Cases - Business
Invoice Understanding Purchase Order Understanding Contract Understanding
Material intended for printing is often multiple columns as it improves human reading performance and provides flexibility in layout to insert images and tables without wasting printable space.
For Marmot: The only description I could find of the dataset, beyond "2000 page images" was here http://128.84.4.34/pdf/2001.01469 where they mention that 1019 / 2000 page images had tables and https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6195411 where they mention 1349 tables total
Note: TableBank (from MSR – China) is a dataset with 417k total labelled example pages (417k pages with table border, 145k pages with table structure). TableBank table structures only have the logical coordinates for the tables identified, specifically the rowSpan and colSpans for each cell along with the text content. TableBank does not provide the bounding box coordinates in the image for tables boundary or each table cell, which an object-detection network would require for training.
Originally TableBank ,was released under CC-Attribution-NonCommercial-NoDerivs License (which is very restrictive). However, in May 2021, now changed to Apache-2.0 license from repo.