New features added to the ADF service last week make handling flexible schemas and schema drift scenarios super easy when construction Mapping Data Flows (public preview) for data transformation at scale:
- New column pattern capabilities added to address columns by position and stream names. This is in addition to the existing features to match columns by name or by data type. See documentation here for schema drift options.
- Automatically skip duplicate column mappings by using new options for skip duplicates in the Select transformation.
- ADF can now automatically infer data types for newly arriving columns using the infer data type option in Source transformation schema drift.
- We've added Rule-based Mapping features with column pattern matching to make it super easy to create mappings for both static and flexible schemas.
- Rule-based mapping is available in Select and Sink transformations.