World Wide Data Vault Consortium (WWDVC)
Unlocking Global Insights With Invaluable Data Vault Knowledge
Gain a competitive edge with access to past conference videos through our Professional Membership. Explore expert insights and innovative solutions in data management, staying ahead in this dynamic field. Whether you’re a seasoned pro or new to the game, these videos provide valuable knowledge to boost your career and enhance your organization’s data strategy. Don’t miss out – unlock the power of data today!
About Your Access
- Full 12 Month Access
- Review Content Any Time
- High Quality Video Delivery
A yearly subscription to our conference videos featuring Data Vault thought leaders provides exceptional value, offering access to cutting-edge insights and strategies. Stay at the forefront of data management and continuously enhance your knowledge and skills. Invest in your professional growth with a Professional Membership today!
Conference Details
Welcome to WWDVC 2019 – World Wide Data Vault Consortium
We hold this conference yearly, in 2019 we had over 112 attendees, 15 presentations including 3 deep dive hands on sessions. Our presenters spoke about everything from Automation, to virtualization, to Data Vault in the cloud. Included in these presentations were business value presentations, culture change, and test-automation! These presentations are just as groundbreaking today as the day they were recorded.Not a Member Yet?
Unlock the power of knowledge and transform your data management expertise with our conference membership. Gain exclusive access to invaluable insights, strategies, and innovations shared by top industry leaders.
Don’t miss out – seize this opportunity to supercharge your professional growth and gain the competitive edge with your conference membership!
Check out these incredible numbers
The amount of knowledge packed into these sessions is phenomenal. Don't miss this opportunity to learn fantastic new ideas.
Conference Sessions
Brainstorming Session (Dan Linstedt, Sanjay Pande)
[student]
Slides: Monday-01-Brainstorming
[/student]
Welcome to our Brain Storming Session
you can introduce yourself to fellow attendees, find out what Data Vault is all about, bring your most complex questions for a Q&A.
LEAVE COMMENTS, ASK QUESTIONS BELOW!
Below are some steps for brainstorming outlined in Forbes: <– click here for the full article.
1. Lay out the problem you want to solve.
This may be easier said than done. Keeney describes a doctoral student who is at sea while trying to come up with a dissertation topic and advisor. The student grasps for ideas with only the vaguest idea of a goal, stated as negatives rather than positives. “I don’t think I could do it,” “it is not interesting to me,” “it seems too hard,” and “it would be too time consuming.” Then finally someone suggests an idea that doesn’t have any of those negatives. The doctoral student grabs the topic. But Keeney says this is a poor way to make a major decision. Instead the student should keep pushing until they come up with at least five more alternatives, and then, considering all those, “identify your objectives for your dissertation, evaluate the alternatives and select the best.” It will be well worth the effort.
2. Identify the objectives of a possible solution.
This is what Keeney did for the German energy company and what he’s done for several government agencies, including the Department of Homeland Security and the energy department. It’s not easy and it takes time but if you can approach your goals critically and hone in on what you want to achieve, your brainstorming session will be much more effective.
3. Try to generate solutions individually.
Before heading into a group brainstorming session, organizations should insist that staffers first try to come up with their own solutions. One problem with group brainstorming is that when we hear someone else’s solution to a problem, we tend to see it as what Keeney calls an “anchor.” In other words, we get stuck on that objective and potential solution to the exclusion of other goals.
4. Once you have gotten clear on your problems, your objectives and your personal solutions to the problems, work as a group.
Though he acknowledges that it’s a challenge not to “anchor” on one solution in a brainstorming session, Keeney believes that if participants have done their homework, clarifying the problem, identifying objectives, and individually trying to come up with solutions, a brainstorming session can be extremely productive.
Panel Discussion and Q&A
No Slides for Download - Please Leave Comments Below
For the first time ever we will invite speakers to participate in a panel discussion followed by a Q&A session.
We will pose questions to the speakers that you want answers to, they can be about their projects, or about their tooling, environments and more! We chose the most requested questions for the panel discussion, and had some really interesting discussions!
Data Vault Games
Sorry, we don't have any videos for Data Vault Games, also - No Slides for Download.
Please leave comments and feedback below, so that we know how to improve the games next year - OR suggest new games for next year.
Our Authorized Trainer: Cindi Meyersohn will lead you through engaging games…
This was so much fun, EVERYONE requested it for 2020!
Conference Kick Off
Speaker: Dan Linstedt
I will welcome everyone to the conference, and kick it off, by introducing our first keynote speaker. This year’s conference has moved downstairs so we can accommodate more attendees, and provide the upstairs room as a permanent vendor floor.
Slides for Download: Wednesday-01-Kickoff
Keynote - Double Vision or Depth Perception?
Speaker: Eric Kavanagh
When you know better, you do better. That’s been the driving force for data warehousing since its inception. Today, despite some rumors of its imminent demise, data warehousing flourishes. There are more solutions, users and ideas than ever, and the waves of innovation keep coming. But what’s the role of so-called Artificial Intelligence? Will AI supplant, disrupt or augment DW?
The simple answer is… yes. From the core practice of data modeling, to the outermost edges of the intelligence arc, AI will infuse itself throughout the practice and theory of data warehousing. If joined harmoniously, this new nexus will help organizations avoid blind spots, gain invaluable depth perception, and see more clearly the world of opportunities and threats. If done poorly, the result will be double vision, vertigo and bad decisions.
Slides for Download:Wednesday-02-EDW and AI
Automated Testing in a Data Vault
Speaker: Neil Strange, Business Thinking
So what can you test in a Data Vault 2.0 project when you have automation? And why not automate the testing as well while you’re at it?
Testing a data warehouse is a very different sort of challenge than a regular software project – there is a lot more data, there is a lot of repetition, and functionality tends to be relatively straightforward.
This presentation will provide practical demonstrations of testing in action as well as suggest a practical, agile testing framework, that works, for your next Data Vault 2.0 project.
What you will learn:
- An approach to automated testing for Data Vault 2.0 projects.
- A framework for functional, data and smoke tests.
- Integrating testing with a metadata mart.
Slides for Download: Wednesday-04-Automated-Testing
When Murphy Strikes
Speaker: Nols Ebersohn, Certus Solutions
Our world is governed by laws. Natural laws – what goes up must come down. Laws of thermo-dynamics. Laws of unintended consequences. Murphy’s law – If things could go wrong, they will go wrong at some point (Usually when you can least afford it to go wrong). These laws will play out in your organisation and in your Data Vault.
Ever had the wrong file/data arrive, invariably only discovered some time later? Data arrive out of sequence, invariably only discovered some time later? Require to load history pre-dating the current already time variant data in your DV satellite? In other words, do you operate in the real world?
We know and understand that the Data Vault standard processes are quite forgiving in nature. For example – late arriving keys, broken referential integrity and the like are ingested with no ill effect to the Data Vault. However the above scenarios traditionally could perhaps force you to roll back to a certain point (if you can) and replay the files or data streams in the correct order (if possible). If neither of these options are available – a hand crafted custom jobs would be required to go and “fix” the issues. This means that you will require special permission and sign-off to correct any incident (Murphy’s law) , compromise the audit trial of the data within your DV and delay the availability of the data while you fix and rigorously test the fix to ensure nothing else is compromised (Law of unintended consequences)!
This paper will explore an extended use of the Record Tracking satellite construct to allow the above scenarios to be remedied without the need to replay data into the DV. In doing so the audit trial of information will remain in-tact and provide little to no delay to information delivery to the business.
What you will learn:
- Advanced Techniques for Operational Resilience for your Data Vault
How to build a Fast & Good Ontology
Speaker: John Giles, Country Eye
I love Dan Linstedt’s 2016 blog on models, business purpose, and data as an asset. I unreservedly back his call for people who can get their hands on an enterprise-level “ontology” to focus on it while building a Data Vault solution. His warning if you don’t? “… the full value of the raw Data Vault cannot be realised.”
Unfortunately, this might be where you struggle. You may ask, “What on earth is an ‘ontology’, because I won’t know if I’ve got one if I don’t know what I’m looking for?” Or you might find one but conclude it’s not helpful – I’ve had clients mandate that their enterprise model is so bad it must not be used to drive their Data Vault initiative. Or maybe you can’t find one at all (good or bad), but business pressures mean you can’t spend months or years building your own ontology. And even if you are lucky enough to get your hands on a suitable ontology, what do you do with it? Lots of questions!
This session lays the foundations for Data Vault model success. Participant interaction will allow the “before your eyes” embryonic construction of an enterprise ontology, using a proven approach. It may surprise you how quickly a “sufficient” model can be assembled from scratch (or an existing ontology refactored to reflect best practices). The approach engages essential business people alongside the technical people in the journey. And it can be fun!!
What you will learn:
- Demystification of the term “enterprise ontology”
- Why and when to build a top-down enterprise view
- The approach to fast-&-good building of your own enterprise ontology (or refactoring the one you wish you hadn’t inherited!)
- Pointers to some jump-start resources
Slides for Download: Wednesday-06-Fast-Food-Ontology
Accelerating DV2.0 With Automation
Speaker: Kevin Marshbank, Senior Solutions Architect, WhereScape
WhereScape® Data Vault Express™ helps IT teams each day make the implementation of Data Vault 2.0 a practical reality through automation. In this session, hear how WhereScape Data Vault Express has evolved in its capabilities since its initial release and where it is headed next. Hear of some of the successes that customers using the data vault automation software are having and what has been key to their ability to implement.
As a WhereScape senior solutions architect, Kevin has helped people and organizations quickly automate their data warehouse initiatives, from build-out to continuous improvement and maintenance. Kevin teaches teams to work in an agile method while increasing engagement with the business and greatly improving their ability to leverage one of their greatest assets…their data.
Slides for Download: Wednesday-07-AcceleratingDV2
Why Data Vault in the Digital Era?
Speaker: Mary Mink, Chestnut Hill Technologies
The importance of data has grown exponentially over the past decade and with it the amount of data we are generating. We have gone from utilizing data for daily operational reporting needs to a Digital Era requiring near real-time availability of information. Hospitals depend on it to save lives; NASA uses it to enable space travel and we use it every day to guide us to find the best driving routes.
Utilizing extremely large volumes of data at high speeds is required to support the emerging technology needs which have become key drivers in our Data-Centric world. These drivers include the following…
- Mobile Application Analytics
- Smart Meters
- Sensors for Monitoring
- Internet of Things (IoT)
- Machine Learning
- Data Science (Predictive, Prescriptive and Descriptive) Modeling
- Artificial Intelligence
- Self-Service Business Intelligence and Data Visualizations
- Data-as-a-Service (DaaS)
Our Technology has advanced so much providing us options for ways to ingest massive amounts of data at high speeds. But what do you do with the data once you get it? How do you make the data actionable in a near-real-time basis? How do you manage and govern it?
What you will learn:
In this session, you will learn why you need Data Vault 2.0 to help you handle the main data challenges in the Digital Era. The following topics will be discussed.
Top 10 Data Challenges
- Data Ingestion of Big Data
- Data Integration of hybrid sources while providing a single version of the truth
- Data Latency and Timing including batch, event-driven and streaming
- Agility of adding new data sources
- Resilience to evolving business needs
- Scalability for volume growth and expansion of data
- Performance of both ingestion and consumption of data
- Quality and Governance of data
- Auditability for compliance
- Management and Support of the data processes
After this session, you will know what the main benefits of using Data Vault 2.0 are and how you can help your company to increase revenues, decrease costs, minimize risk and enhance customer experience thru utilization of data best practices. You will have the approach to deliver the “Right data at the right time!”.
Slides for Download: Thursday-02-Why-Choose-DataVault
Keynote - Modernized Data Architecture
Speaker: WH (Bill) Inmon, Forest Rim
What You Will Learn:
- developments in managing voice to text transcription
- how context can be incorporated into sentiment analysis
- how the dimensional model can be streamlined
Slides for Download: Thursday-01-Keynote-Modern-Data-Architecture
Stream Processing into Data Vault 2.0
Speaker: Bruce McCartney
The purpose of this presentation is to review various alternative architectures, methods and tools for streaming near real-time data into the Data Vault 2.0 system of business Intelligence. In addition, using streaming processing for Data Lake provisioning will be discussed – with recommendations for fitting Data Vault 2.0 concepts with a successful data lake.
The presentation will also give attendees a good understanding of the basics required to accomplish streaming and exposure to several different development paradigms available. The focus will be in using SQL stream processing to accomplish this, and give a customer example of an approach taken.
What you will learn:
- architecture for streaming data
- raw data vault considerations of near real-time data
- business vault alternatives for streaming
Slides for Download: Thursday-04-Streaming-Data-Into-DV
Overcoming Uncertainty: Student Led Storytelling
Speakers: Daniel Hartness, Sean Duffy, Western Carolina University
We will demonstrate how we use Data Vault 2.0 at a university level to meet the requirements of a rapidly changing community. The North Carolina Data Dashboard is a student-lead project which hosts a wide variety of county-level and industry-by-county-level data for the region, informing users with easily digestible data displays of economic indicators. Data Vault 2.0 has created opportunities for expansive scalability in this project. We began by hosting data only for Western North Carolina, but have steadily grown to hosting data, such as labor markets, product markets, real estate and workforce demographics for the entire state. The Data Vault 2.0 Framework was critical in allowing us to implement new data series with only minor modifications. Because of this, we are now capable of scaling our databases to house nationwide economic data with relative ease.
Data Vault 2.0 was further pivotal in helping us solve a problem with a high turnover rate as students continuously join and leave the project upon graduation. New students with little database design experience are able to grasp our model due to the simplicity of the three basic structures. Furthermore, Data Vault 2.0 Methodology structures our User Guide so that our model has higher probability to mature. The process of updating and obtaining new data series is agile for new students because of the architecture. This data is then moved to our persistent-storage database where it is organized into hub, link, and satellite elements. Views are then created from these elements to form our access layer. The structure of the access area allows for the formation of views that fit the needs of Tableau. It also allows for future capability such as the support to directly connect to an in-house Tableau Server. Overall, Data
What you will Learn:
- How to implement the Data Vault 2.0 Model into a university/student-led environment.
- How to extend into new data sources with the Data Vault 2.0 Architecture.
- How Data Vault 2.0 can be used to overcome short time frames in a rapidly-changing community.
- How to use Data Vault 2.0 to tell a story.
Slides for Download: Thursday-03-Overcoming-Uncertainty
Data Vault 2.0 On MongoDB
Speaker: Michael Olschimke
MongoDB is a popular document-oriented NoSQL database. It is often used for operational systems, but can be used for analytical purposes very well, too, due to support for aggregations. Because of growing market demand for data vault solutions on MongoDB, Scalefree worked on implementing a Data Vault model based on MongoDB collections. From that, we derived a presentation layer and used the MongoDB BI connector to bring the information to a dashboard.
What you will learn:
- What is MongoDB
- How to load Data Vault Collections
- What is our model approach to nested document structures
- How do Hub, Link and Satellite documents look like on MongoDB
- How is the data presented using MongoDBs BI connector
Slides for Download: Thursday-05-DataVault-on-MongoDB
Data Vault in the Cloud! Mind Blown!
Speaker: Kent Graziano
We all know that data warehouses and best practices for them are changing dramatically today. As organizations build new data warehouses and modernize their data analytics ecosystem, they are turning to cloud-based data warehousing services in hopes of taking advantage of the performance, elasticity, concurrency, simplicity, and lower cost of a cloud-based platform or simply to reduce their data center footprint (and the maintenance that goes with that). Data Vault 2.0 gives us a system for designing and deploying highly flexible, scalable, and adaptable data structures for enterprise data warehouse repositories. So is there any additional benefits a Data Vault 2.0 warehouse can get from moving to the cloud? In this talk I will introduce some unique capabilities provided in the cloud, as evidenced by Snowflake, that can be leveraged to make your Data Vault 2.0 system more agile, flexible, and scalable.
What will be presented:
Applying Snowflake features to do the following:
- Hyper-parallelize and automatically scale load processes
- Easily include JSON in your Data Vault
- Load using multi-table inserts
- Easily separate PII data
- Use MD5 hash function
- Virtualize your Information Marts
Slides for Download: Thursday-06-DataVault-in-the-Cloud
Operational Excellence & Best Practices
Speaker: Gregory Locke
This talk will cover the following topics:
- Business Layer Stabilization
- How to insulate your customers from the back-end load process and provide 24/7 access to an active vault.
- In one vault, our business layer supports web services and direct reads from downstream systems. In another vault, the business layer supports operational reporting.
- We chose to implement an active/standby copy of the business layer for the data distribution hub. One copy is online while the other is off-line during the vault update.
- This same architecture supports operational reporting use-cases and will also support a data mart analytics use-cases.
- How did we do this?
- Process Scheduling and Monitoring
- How to integrate operational scheduling into your load process.
- Whether you need to create your own delta feeds or run a re-load, fine grained run-time controls make it easy to manage the process.
- Full or partial re-loads.
- Performance monitoring and the importance of job logs
- Exploiting pattern based ingestion with re-usable components
- Finding the patterns – Identifying re-usable components and when enough is enough.
- Embed intelligent operational process controls – how to re-load, re-start or re-run without worries
- How to design for failures –
- Table driven parameter based run-time controls
What you will learn:
- How to insulate your customers from the back-end load process and provide 24/7 access to an active vault.
- How to integrate operational scheduling into your load process.
- How to identify re-usable components and when enough is enough.
- How to embed intelligent operational controls in the load process.
Slides for Download:Friday-01-Operational-Excellence
Tricks, Tips, and Tools for Test Automation
Speaker: France Pare
It’s been two years since Intact’s Data & Analytics department started building a Business Intelligence toolkit (BI toolkit) to acquire data, improve quality and make data available to end users faster. Chiming into DevOps, the team first built a test automation component and then integrated it with code generation and continuous integration/deployment.
In the first year, test automation was designed and built. It was later integrated with code generation, continuous integration and delivery.
As we pushed forward it became very apparent that project teams needed to change the way they worked. Tools were introduced, which challenged how projects did work, including Requirements gathering, Mapping, Development. Implementing the data vault, which is pattern based, along with tools like WhereScape changed our Software Development Life Cycle and the testing paradigm.
What does it mean to automate testing in the data warehouse? Do we need to test the code generated? What quality controls should be put in place to detect defects? Are there special circumstances when additional testing or manual work will be needed? How much data is needed to validate things are working properly and correctly?
The quality controls put in place to monitor the ETL/ELT processes were the first step to ensure the quality of the data in our Data Vault Data Warehouse. The automation framework reduced the risk of introducing defect, but we discovered the need to have additional quality controls in place to ensure that the automation process is defect free. Let’s share the lessons learned in our journey with you
What we will cover:
- Data Warehouse automation context: Pro’s and con’s
- People: New skills and paradigm
- Process: Assumptions, standards and pattern based
- Tools: Testing toolkit, Jenkins, Wherescape, Git and more
- Quality control: Automated vs manual code
- Demo
- Lesson learned: Do and don’t
- What’s next? More automation, Dashboard, Test data management
Slides for Download:Friday-02-Test-Automation
Data Vault Security
Speakers: Troy Belanger, Mike Magalsky
Application of security in the Data Vault
Examples of securing data at the data layer for visualizations with BI Tools without adding or buying additional software.
Determining Classifications for a Health Care Client and then implementing them
What you will learn:
- Application of security in the Data Vault
- Data Security Assessment Process
- Data Provisioning Challenges
- Problem: Set-based Data Provisioning=A fast road to chaos
- Answer: Policy-based Data Provisioning = A manageable solution
Slides for Download:Friday-03-DataVault-Security
Conference Closing Remarks
Speaker: Dan Linstedt
Once again, time to say good bye, and thank you for coming to the conference. I will do my conference wrap-up, and returning this year will be the “funniest quotes” from the sessions. If you hear a speaker say something funny, TWEET IT with #WWDVC 2019
Thanks For Coming and See you NEXT YEAR!
Slides for Download:Friday-04-Closing