In a previous article we explained the syntax used to write Drupal migration. We also provided references of subfields and content entities' properties including those provided by the Commerce module. This time we are going to list the configuration options of many migrate source plugins. For example, when importing from a JSON file you need to specify which data fetcher and parser to use. In the case of CSV migrations, the source plugin configuration changes depending on the presence of a headers row. Finding out which options are available might require some Drupal development knowledge. To make the process easier, in today’s article we are presenting a reference of available configuration options for migrate source plugins provided by Drupal core and some contributed modules.

For each migrate source plugin we will present: the module that provides it, the class that defines it, the class that the plugin extends, and any inherited options from the class hierarchy. For each plugin configuration option we will list its name, type, a description, and a note if it is optional.
Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Extends: Drupal\Core\Plugin\PluginBase
This abstract class is extended by most migrate source plugins. This means that the provided configuration keys apply to any source plugin extending it.
List of configuration keys:
The high_water_property and track_changes are mutually exclusive. They are both designed to conditionally import new or updated records from the source. Hence, only one can be configured per migration definition file.
Module: Migrate (Drupal Core)
Class: Drupal\migrate\Plugin\migrate\source\SqlBase
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This abstract class is extended by migrate source plugins whose data may be fetched via a database connection. This means that the provided configuration keys apply to any source plugin extending it.
In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:
To explain how these configuration keys are used, consider the following database connections:
<?php $databases['default']['default'] = [ 'database' => 'drupal-8-or-9-database-name', 'username' => 'drupal-8-or-9-database-username', 'password' => 'drupal-8-or-9-database-password', 'host' => 'drupal-8-or-9-database-server', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ]; $databases['migrate']['default'] = [ 'database' => 'drupal-6-or-7-database-name', 'username' => 'drupal-6-or-7-database-username', 'password' => 'drupal-6-or-7-database-password', 'host' => 'drupal-6-or-7-database-server', 'port' => '3306', 'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql', 'driver' => 'mysql', ];
This snippet can be added to settings.php or settings.local.php. The $databases array is a nested array of at least three levels. The first level defines the database keys: default and migrate in our example. The second level defines the database targets: default in both cases. The third level is an array with connection details for each key/target combination. This documentation page contains more information about database configuration.
Based on the specified configuration values, this is how the Migrate API determines which database connection to use:
Note that all values configuration keys are optional. If none is set, the plugin will default to use the connection specified under $databases['migrate']['default']. At least, set the key configuration even if the value is migrate. This would make it explicit which connection is being used.
Module: Migrate Drupal (Drupal Core)
Class: Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase
Extends: Drupal\migrate\Plugin\migrate\source\SqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, and ignore_map.
This abstract class provides general purpose helper methods that are commonly needed when writing source plugins that use a Drupal database as a source. For example, check if the given module exists and read Drupal configuration variables. Check the linked class documentation for more available methods.
In addition to the keys provided in the parent class chain, this abstract class provides the following configuration key:
Warning: A plugin extending this abstract class might want to use this configuration key in the source definition to set module dependencies. If so, the expected keys might clash with other source constants used in the process pipeline. Arrays keys in PHP are case sensitive. Using uppercase in custom source constants might avoid this clash, but it is preferred to use a different name to avoid confusion.
This abstract class is extended by dozens of core classes that provide an upgrade path from Drupal 6 and 7. It is also used by the Commerce Migrate module to read product types, product display types, and shipping flat rates from a Commerce 1 database. The same module follows a similar approach to read data from an Ubercart database. The Paragraphs module also extends it to add and implement Configurable Plugin interface so it can import field collection types and paragraphs types from Drupal 7.
Module: Migrate Drupal (Drupal Core). Plugin ID: d8_config
Class: Drupal\migrate_drupal\Plugin\migrate\source\d8\Config
Extends: Drupal\migrate_drupal\Plugin\migrate\source\DrupalSqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, ignore_map, and constants.
This plugin allows reading configuration values from a Drupal 8 site by reading its config table.
In addition to the keys provided in the parent class chain, this plugin does not define extra configuration keys. And example configuration for this plugin would be:
source: plugin: d8_config key: migrate skip_count: true
In this case we are setting the key property from SqlBase to use the migrate default database connection. The skip_count from SourcePluginBase indicates that there is no need to count how many records exist in the source database before executing migration operations like importing them.
This plugin is presented to show that Drupal core already offers a way to migrate data from Drupal 8. Remember that there are dozens of other plugins extending DrupalSqlBase. It would be impractical to list them all here. See this API page for a list of all of them.
Module: Migrate Source CSV. Plugin ID: csv
Class: Drupal\migrate_source_csv\Plugin\migrate\source\CSV
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This plugin allows reading data from a CSV file. We used this plugin in the CSV migration example of the 31 days of migration series.
In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:
Important: The configuration options changed significantly between the 8.x-3.x and 8.x-2.x branches. Refer to this change record for a reference of how to configure the plugin for the 8.x-2.x.
For reference, below is the source plugin configuration used in the CSV migration example:
source: plugin: csv path: modules/custom/ud_migrations/ud_migrations_csv_source/sources/udm_photos.csv ids: [photo_id] header_offset: null fields: - name: photo_id label: 'Photo ID' - name: photo_url label: 'Photo URL'
Module: Migrate Spreadsheet. Plugin ID: spreadsheet
Class: Drupal\migrate_spreadsheet\Plugin\migrate\source\Spreadsheet
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This plugin allows reading data from Microsoft Excel and LibreOffice Calc files. It requires the PhpOffice/PhpSpreadsheet library and many PHP extensions including ext-zip. Check this page for a full list of dependencies. We used this plugin in the spreadsheet migration examples of the 31 days of migration series.
In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:
Note that nowhere in the plugin configuration you specify the file type. The same setup applies for both Microsoft Excel and LibreOffice Calc files. The library will take care of detecting and validating the proper type.
For reference, below is the source plugin configuration used in the LibreOffice Calc migration example:
source: plugin: spreadsheet file: modules/custom/ud_migrations/ud_migrations_sheets_sources/sources/udm_book_paragraph.ods worksheet: 'UD Example Sheet' header_row: 1 origin: A2 columns: - book_id - book_title - 'Book author' row_index_column: 'Document Row Index' keys: book_id: type: string
Module: Migrate Plus
Class: Drupal\migrate_plus\Plugin\migrate\source\SourcePluginExtension
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This abstract class provides extra configuration keys. It is extended by the URL plugin (explained later) and by source plugins provided by other modules like Feeds Migrate.
In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:
See the code snippet for the Url plugin in the next section for an example of how these configuration options are used.
Module: Migrate Plus. Plugin ID: url
Class: Drupal\migrate_plus\Plugin\migrate\source\Url
Extends: Drupal\migrate_plus\Plugin\migrate\source\SourcePluginExtension
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, fields, and ids.
This plugin allows reading data from URLs. Using data parser plugins it is possible to fetch data from JSON, XML, SOAP, and Google Sheets. Note that this source plugin uses other plugins provided by Migrate Plus that might require extra configuration keys in addition to the ones explicitly defined in the plugin class. Those will also be listed.
In addition to the keys provided in the parent class chain, this plugin provides the following configuration keys:
The data parser plugins provide the following configuration keys:
The HTTP data fetcher plugins provide the following configuration keys:
The basic and digest authentication plugins provide the following configuration keys:
The OAuth2 authentication plugin requires the sainsburys/guzzle-oauth2-plugin composer package to work. It provides the following configuration keys:
The client credentials grant type requires the following configuration keys:
For configuration keys required by other grant types, refer to the classes that implement them. Read this article on adding HTTP request headers and authentication parameters for example configurations.
There are many combinations possible to configure this plugin. In the 31 days of migration series there are many example configurations. For reference, below is the source plugin configuration used in the local JSON node migration example:
source: plugin: url data_fetcher_plugin: file data_parser_plugin: json urls: - modules/custom/ud_migrations/ud_migrations_json_source/sources/udm_data.json item_selector: /data/udm_people fields: - name: src_unique_id label: 'Unique ID' selector: unique_id - name: src_name label: 'Name' selector: name - name: src_photo_file label: 'Photo ID' selector: photo_file - name: src_book_ref label: 'Book paragraph ID' selector: book_ref ids: src_unique_id: type: integer
Module: Migrate (Drupal Core). Plugin ID: embedded_data
Class: Drupal\migrate\Plugin\migrate\source\EmbeddedDataSource
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This plugin allows the definition of data to be imported right inside the migration definition file. We used this plugin in many of the examples of the 31 days of migration series. It is also used in many core tests for the Migrate API itself.
In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:
Many examples of 31 days of migration series use this plugin. You can get the example modules from this repository. For reference, below is the source plugin configuration used in the first migration example:
source: plugin: embedded_data data_rows: - unique_id: 1 creative_title: 'The versatility of Drupal fields' engaging_content: 'Fields are Drupal''s atomic data storage mechanism...' - unique_id: 2 creative_title: 'What is a view in Drupal? How do they work?' engaging_content: 'In Drupal, a view is a listing of information. It can a list of nodes, users, comments, taxonomy terms, files, etc...' ids: unique_id: type: integer
This plugin can also be used to create default content when the data is known in advance. We often present Drupal site building workshops. To save time, we use this plugin to create nodes which are later used when explaining how to create Views. Check this repository for an example of this. Note that it uses a different directory structure to store the migrations as explained in this blog post.
Module: Migrate Drupal (Drupal Core). Plugin ID: content_entity
Class: Drupal\migrate_drupal\Plugin\migrate\source\ContentEntity
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This plugin returns content entities from a Drupal 8 or 9 installation. It uses the Entity API to get the data to migrate. If the source entity type has custom field storage fields or computed fields, this class will need to be extended and the new class will need to load/calculate the values for those fields.
In addition to the keys provided in the parent class chain, this plugin provides the following configuration key:
For reference, this is how this plugin is configured to get all nodes of type article in their default language only:
source: plugin: content_entity:node bundle: article include_translations: false
Note: this plugin was brought into core in this issue copied from the Drupal 8 migration (source) module. The latter can be used if the source database does not use the default connection.
Module: Migrate Plus. Plugin ID: table
Class: Drupal\migrate_plus\Plugin\migrate\source\Table
Extends: Drupal\migrate\Plugin\migrate\source\SqlBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, source_module, key, target, database_state_key, batch_size, and ignore_map.
This plugin allows reading data from a single database table. It uses one of the database connections for the site as defined by the options. See this test for an example on how to use this plugin.
In addition to the keys provided in the parent class chain, this plugin provides the following configuration key:
Module: Migrate (Drupal Core). Plugin ID: empty
Class: Drupal\migrate\Plugin\migrate\source\EmptySource
Extends: Drupal\migrate\Plugin\migrate\source\SourcePluginBase
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
This plugin returns an empty row by default. It can be used as a placeholder to defer setting the source plugin to a deriver. An example of this can be seen in the migrations for Drupal 6 and Drupal 7 entity reference translations. In both cases, the source plugin will be determined by the EntityReferenceTranslationDeriver.
In addition to the keys provided in the parent class chain, this plugin does not define extra configuration keys. If the plugin is used with source constants, a single row containing the constant values will be returned. For example:
source: plugin: empty constants: entity_type: node field_name: body
The plugin will return a single row containing 'entity_type' and 'field_name' elements, with values of 'node' and 'body', respectively. This is not very useful. For the most part, the plugin is used to defer the definition to a deriver as mentioned before.
Module: Migrate Drupal (Drupal Core). Plugin ID: md_empty
Class: Drupal\migrate_drupal\Plugin\migrate\source\EmptySource
Extends: Drupal\migrate\Plugin\migrate\source\EmptySource
Inherited configuration options: skip_count, cache_counts, cache_key, track_changes, high_water_property, and source_module.
By default, this plugin returns an empty row with Drupal specific config dependencies. If the plugin is used with source constants, a single row containing the constant values will be returned. These can be seen in the user_picture_field.yml and d6_upload_field.yml migrations.
In addition to the keys provided in the parent class chain, this abstract class provides the following configuration keys:
In Drupal core itself there are more than 100 migrate source plugins, most of which come from the Migrate Drupal module. And many more are made available by contributed modules. It would be impractical to document them all here. To get a list by yourself, load the plugin.manager.migrate.source service and call its getFieldStorageDefinitions() method. This will return all migrate source plugins provided by the modules that are currently enabled on the site. This Drush command would get the list:
# List of migrate source plugin definitions. $ drush php:eval "print_r(\Drupal::service('plugin.manager.migrate.source')->getDefinitions());" # List of migrate source plugin ids. $ drush php:eval "print_r(array_keys(\Drupal::service('plugin.manager.migrate.source')->getDefinitions()));"
To find out which configuration options are available for any source plugin consider the following:
What did you learn in today’s article? Did you know that migrate source plugins can inherit configuration keys from their class hierarchy? Were you aware that there are so many source plugins? Other than the ones listed here, which source plugins have you used? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.
At Agaric, we perform a lot of Drupal upgrades. These very often involve transitioning away from older versions of PHP. Even when your hosting service provides multiple versions of PHP, you can still run into issues activating the appropriate one for each site: whether that's within the web server, at the command line, or via Drush (or other tools). In this blog post, we'll be providing remedies for all of these cases. Most content applies to any PHP application.
The National Institute for Children’s Health Quality (NICHQ) delivers a steady stream of demonstrated best practices to help children and their families thrive. Many of these findings come out of the work of NICHQ’s varied collaborative improvement initiatives, in which clinical, research, and community-based teams come together around targeted improvements (e.g., strengthening maternity care services in hospitals) or introducing broad systems-level change (e.g., coordinating early childhood systems).

We live amidst a Digital Commons - technology that is built with the principles of freedom and transparency baked into its code and design. It's maintained out in the open by the free software community. This commons is invisible to many of us, but the closer we are to the technology we use, the more that it comes into focus. At Agaric we are knee deep in this Digital Commons. Our name Agaric is a nod to the mycelial nature of the open web. We help create, maintain, and promote free and open-source software that make up this commons.
Resources can be either a file-based resource such as a PDF or a link-based resource such as a url to a website. Resources can also be categorized by resource type and can also have an associated tag. An associated image such as the cover of a PDF resource or a logo or screenshot from a website link can make the resource more visually attractive in various displays.
The resource content type lets you post a PDF (or various other text based formats) so that site visitors can easily view and download. The body field lets you provide some information about the resource to help site visitors determine the usefulness of the resource before viewing.
Helpful tips include ensuring that your PDF has a user-friendly machine name (and/or using the file description field for the title users will see) and adding a cover image for example, in the image field, to provide a visual clue to the resource.
For link-based resources, you'll need to paste in the full website URL, such as https://drutopia.org. The Link text could read "Visit the Drutopia website."
(Technically, there is one formatter each for text lists and for entity references, but all the options which In Other Words adds are the same for both.)
We can configure a separator between list items, a final join word, and whether to connect the final items with a comma (or semicolon or any other punctuation we can imagine for our list).
In all its glory in context:
A program or application is free software when it is intentionally licensed with our freedom in mind. Specifically, four freedoms (numbered using the zero-based numbering in programming languages):
From Linux to Firefox to Drupal, free software is a tremendous force in our world. By building free software, we help building a software commons we can all trust and benefit from.
There are similar terms used to describe similar software. Libre software is a synonym for free software. By using the word libre, we remove the ambiguity inherent in free, which can mean free as in no money or free as in liberty. We appreciate this term and use it interchangeably with free software.
Open source is very similar to free software. It emerged, like libre software, as a way to remove ambiguity. However, in doing so it loses the ethical and political power that comes with free/libre software. So while we respect and use the term open source, we prefer free software because of it emphasizes the value of freedom.
The following are excellent pieces for further information on the meaning of free software and some of the differences and debates between the various terms.
Agaric Technology Collective worker-owners have developed, maintained, and upgraded scores of Drupal sites; contributed to more than one hundred modules; and authored authoritative guides and documentation, including The Definitive Guide to Drupal 7 and the aforementioned 31 Days of Drupal Migrations.
We love free and open formats. RSS or Really Simple Syndication is a great way to keep in touch with your favorite blogs and websites.
We have two available:
If you're new to RSS, here's what you need to do:
There are a lot out there. If you'd like to use a web-based reader, one option is CommaFeed. It's free software, easy to use and has lots of features.
If you prefer a desktop application, Raven Reader is also free software and well designed.
Log in with a service provider account
Share information about your organization and post events and services.
Code with confidence. Deploy without doubt. End-to-end testing provides you with the assurance that improvements to your site are not breaking anything else. And when changes do cause regressions, end-to-end tests point you to what needs to be fixed.
Behat makes it possible to set up end-to-end tests that can access Drupal's APIs for setup, context provisioning, and teardown. And it does this with tests that can be read by project managers, product owners, and everybody else involved in the project, because each step is written in plain English.
This training will give you the setup that makes testing a realistic part of your day-to-day development.