From the course: AI Data Strategy: Data Procurement and Storage

Unlock this course with a free trial

Join today to access over 24,800 courses taught by industry experts.

Best practices for sourcing unstructured data

Best practices for sourcing unstructured data

From the course: AI Data Strategy: Data Procurement and Storage

Start my 1-month free trial Buy for my team

Best practices for sourcing unstructured data

“

- [Instructor] In the last video, you saw how to source structured data for ML models, but generative AI brings us into a whole new world, one where we need vast amounts of unstructured data, like text, images, and code. Apart from the sheer volume, the challenge with generative AI is about finding diverse high quality data that you need to help your models learn and consequently generate meaningful content. Teams often think they need to collect every bit of data they can find, but having the biggest data set doesn't guarantee success. It's about having the right kind of data diversity. Think about a large language model like GPT. It's not just trained on perfectly written books and articles. It must also understand how people actually communicate, everything from formal documents to casual conversations, technical manuals to creative writing. This diversity is what allows the model to generate appropriate responses in different contexts. Remember the star framework we discussed…

Contents