The International Summit of Cooperatives convened in Quebec in 2016. The general message of the conference was that cooperatives are everywhere and one only needs to raise awareness for this idea to spread. That seems to be happening as evidenced by the attendance at this conference - 3000 people from 117+ countries according to the International Co-operative Alliance (ICA.coop) one of the sponsors of #ISCOOP2016.

I have been to many cooperative conferences and events, but this one was very different. From the facility to the attendees, the event had an air of style and conform that went beyond attire. I was swimming in a sea of 70-80% middle-aged men in black suits as far as the eye could see. There were also a few groups I saw that were wearing indigenous dress from India, Nepal, Chile, and Congo. Quebec is a mixture of modern and old culture. There were women in authentic Breton garb serving food in the restaurant we visited for lunch, in Old Quebec City, but they were not represented at the coop summit.
 

The largest sponsors of the Summit were the Canadian Government, Canada Economic Development, ICA International Co-operative Alliance and DesJardins. You can see a full listing of all the cooperative sponsors for the event. The attendees were mostly members of Agriculture and Financial coops, both small and large. When I say large, I am talking thousands of members. The point was brought up and highlighted that the International Co-operative Alliance represents close to one billion individual members. Statistics are calculated using the Alliance's formula based on active subscriptions. The ICA maintains the internationally recognised definition of a co-operative in the Statement on the Co-operative Identity and they represent 272 co-operative federations and organizations in 94 countries as of January 2014. The National Cooperative Bank released its annual report in 2015, listing the nation’s top 100 revenue-earning cooperative businesses. These 100 businesses posted revenue of approximately $243.2 billion.

A dominant presence by DesJardins, with over 6 million members, and the Boston Consulting Group (BCG). In my opinion, the BCG message was depressing and the same old Capitalistic message cloaked in a message of "growth" suggesting that "you must grow in order to prosper!" Luckily some cooperative panelists responded with how irresponsible it is to grow for growth sake. A member of DesJardins told me that he personally thinks the coop has gotten too big and we had a great talk about how empathy training should be available to larger entities. Marc Thomas, a DesJardins member, talked about helping people and how the DesJardins cooperative has made some positive changes for him and his community. They are much more willing to lend to smaller cooperatives and they have a host of connections to nurture a small business in start up phases. They play a role similar to a credit union on a much larger scale and they also have deep ties in the community and a network of cooperatives to connect new ideas to funding opportunities.

According to Howard Brodsky there are 50 cooperatives larger than Facebook. Brodsky is Co-Founder, Chairman, and Chief Executive Officer of CCA Global Partner. He is responsible for creating a cooperative retail powerhouse in the marketplace. 

His message was most similar to all the coop conferences I had attended, yet he was much more vocal about how expansive the coop movement is - we are large and we are everywhere. In my opinion, Howard gets it and understands how empathy and caring figure into the movement. He touched on how this simply will not work if we are not honest and caring in our work. He spoke about how important "Stories" are and how they create bonds. The International Co-op Alliance (ica.coop) has built a wonderful way to share our narratives in this digital era - http://stories.coop

Trebor Scholz, a professor at the New School in NYC, and Nathan Schneider, a professor at University of Colorado Boulder, were each on a panel. Nathan's panel was on Multi Sector Activity and he talked about Platform Cooperativism as a way to bring cooperative communities together and how important it is to own the utilities and services we depend on. Trebor on the next panel framed platform coop as a movement and points to the recently published book "Ours to Hack and to Own" as the handbook to get involved in the movement. Copies of book are available from OR Books.

Both panels were lively and got a good response of people talking amongst each other after they ended. Attendees I talked to during the conference were diverse. People from Kenya and the Congo seemed to be the only ones shocked at the implications when I told them about free software. They had never heard of it. People from India that I met either said they knew about it or that they used it in their work. Other people from afar seemed bored and made excuses to not hear about it.

Translation for the speakers and panelists was stellar. No time lag at all and the team was professional and consistent. This did not carry over into the main event participants and attendees. The language barriers seemed to keep people from mingling outside of their party of friends. They sat in groups at the meals and at the sessions. The lunch/dinner seating was round table, with place settings ala extra forks etc. very convenient for conversations. Meals were also used as a venue for a sort of Keynote delivery that happened on several giant screens while the appetizers were served.

The event was full of Pros and positive energy, there were only a few minor Cons:
1. A lot of people in the crowd were unaware of free software, and almost no one used encryption.
2. Most panels were all male - even ones discussing diversity.
3. Some financial coops place most emphasis on growing, as if growth is the only measure of success and value.

Robert Reich got a standing ovation for his keynote with a message that coops are an important part of the business landscape. He spent most of his talk telling an anecdotal touchy feely story with the point that we all need to get along. He seems to be a progressive at times, but he still operates on a lesser of two evils mentality in a two party system - I ask, why won't he support a third party if he is in agreement with most of their platforms?

I spoke to many people throughout the three day event about what Agaric is and what we do. I also talked about Free Software and how it impacts cooperatives and their goals. Some were not aware of free software and the vital role it will play in the future success of cooperatives maintaining autonomy and privacy. I also spoke about Platform Cooperativism and Drutopia as a platform concept. Anyone seeking more information can sign up at the Drutopia.org website to be invited to discussions and have a voice in the process of building a platform cooperative from the ground up.

So, things are looking bright for cooperatives in the future. The cooperative branding and marketing needs building, and the network needs to keep expanding and cross-pollinating. The tireless work and dedication of small groups like the ICA is what makes this all happen. Coops do not need to be large, they need to be nimble and they need to be flexible. With apps like BuyCott it will be much easier to purchase responsibly, buy from cooperatives and support ethical companies. I just got the app and am happily surprised to find out how many dedicated cooperative people there are in the world shopping responsibly already! We can each do our part to make the network stronger and to bring the cooperative movements closer together. How would you bring something cooperative to your community? Even a small local event at your neighborhood coffee shop or a blog post or a Tweet can do a lot to raise the level of awareness of how strong we are together!

The results are in:
Typically an awesome event will end and as time passes, there will be little to no follow-up or tangible results that are published. You wonder if that great project you heard about is flourishing or forgotten. You can see the results of the workshops in Quebec in 2016 and rejoice in the knowledge that we are on our way to autonomy.

Our friend Chuck Bordman has written an excellent blog covering this conference here: Coopmatters.com


Today, we are going to talk about how to manage migrations as configuration entities. This functionality is provided by the Migrate Plus module. First, we will explain the difference between managing migrations as code or configuration. Then, we will show how to convert existing migrations. Finally, we will talk about some important options to include in migration configuration entities. Let’s get started.

Drupal migrations: code or configuration?

So far, we have been managing migrations as code. This is functionality provided out of the box. You write the migration definition file in YAML format. Then, you place it in the migrations directory of your module. If you need to update the migration, you make the modifications to the files and then rebuild caches. More details on the workflow for migrations managed in code can be found in this article.

Migrate Plus offers an alternative to this approach. It allows you to manage migrations as configuration entities. You still use YAML files to write the migration definition files, but their location and workflow is different. They need to be placed in a config/install directory. If you need to update the migration,  you make the modifications to the files and then sync the configuration again. More details on this workflow can be found in this article.

There is one thing worth emphasizing. When managing migrations as code you need access to the file system to update and deploy the changes to the file. This is usually done by developers.  When managing migrations as configuration, you can make updates via the user interface as long as you have permissions to sync the site’s configuration. This is usually done by site administrators. You might still have to modify files depending on how you manage your configuration. But the point is that file system access to update migrations is optional. Although not recommended, you can write, modify, and execute the migrations entirely via the user interface.

Transitioning to configuration entities

To demonstrate how to transition from code to configuration entities, we are going to convert the JSON migration example. You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD config JSON source migration whose machine name is udm_config_json_source. It comes with four migrations: udm_config_json_source_paragraph, udm_config_json_source_image, udm_config_json_source_node_local, and udm_config_json_source_node_remote.

The transition to configuration entities is a two step process. First, move the migration definition files from the migrations folder to a config/install folder. Second, rename the files so that they follow this pattern: migrate_plus.migration.[migration_id].yml. For example: migrate_plus.migration.udm_config_json_source_node_local.yml. And that’s it! Files placed in that directory following that pattern will be synced into Drupal’s active configuration when the module is installed for the first time (only). Note that changes to the files require a new synchronization operation for changes to take effect. Changing the files and rebuilding caches does not update the configuration as it was the case with migrations managed in code.

If you have the Migrate Plus module enabled, it will detect the migrations and you will be able to execute them. You can continue using the Drush commands provided the Migrate Run module. Alternatively, you can install the Migrate Tools module which provides Drush commands for running both types of migrations: code and configuration. Migrate Tools also offers a user interface for executing migrations. This user interface is only for migrations defined as configuration though. It is available at /admin/structure/migrate. For now, you can run the migrations using the following Drush command: drush migrate:import udm_config_json_source_node_local --execute-dependencies.

Note: For executing migrations in the command line, choose between Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided by the two modules have the same name. Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands.

UUIDs for migration configuration entities

When managing migrations as configuration, you can set extra options. Some are exposed by Migrate Plus while others come from Drupal’s configuration management system. Let’s see some examples.

The most important new option is defining a UUID for the migration definition file. This is optional, but adding one will greatly simplify the workflow to update migrations. The UUID is used to keep track of every piece of configuration in the system. When you add new configuration, Drupal will read the UUID value if provided and update that particular piece of configuration. Otherwise, it will create a UUID on the fly, attach it to the configuration definition, and then import it. That is why you want to set a UUID value manually. If changes need to be made, you want to update the same configuration, not create a new one. If no UUID was originally set, you can get the automatically created value by exporting the migration definition. The workflow for this is a bit complicated and error prone so always include a UUID with your migrations. This following snippet shows an example UUID:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'

The UUID a string of 32 hexadecimal digits displayed in 5 groups. Each is separated by hyphens following this pattern: 8-4-4-4-12. In Drupal, two or more pieces of configuration cannot share the same value. Drupal will check the UUID and the type of configuration in sync operations. In this case the type is signaled by the migrate_plus.migration. prefix in the name of the migration definition file.

When using configuration entities, a single migration is identified by two different options. The uuid is used by the Drupal’s configuration system and the id is used by the Migrate API. Always make sure that this combination is kept the same when updating the files and syncing the configuration. Otherwise you might get hard to debug errors. Also, make sure you are importing the proper configuration type. The latter should not be something to worry about unless you utilize the user interface to export or import single configuration items.

If you do not have a UUID in advance for your migration, you can try one of these commands to generate it:

# Use Drupal's UUID service.
$ drush php:eval "echo \Drupal::service('uuid')->generate(). PHP_EOL;"

# Use a Drush command provided by the Devel module, if enabled.
$ drush devel:uuid

# Use a tool provided by your operating system, if available.
$ uuidgen

Alternatively, you can search online for UUID v4 generators. There are many available.

Technical note: Drupal uses UUID v4 (RFC 4122 section 4.4) values which are generated by the `uuid` service. There is a separate class for validation purposes. Drupal might override the UUID service to use the most efficient generation method available. This could be using a PECL extension or a COM implementation for Windows.

Automatically deleting migration configuration entities

By default, configuration remains in the system even if the module that added it gets uninstalled. This can cause problems if your migration depends on custom migration plugins provided by your module. It is possible to enforce that migration entities get removed when your custom module is uninstalled. To do this, you leverage the dependencies option provided by Drupal’s configuration management system. The following snippet shows how to do it:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
  enforced:
    module:
      - ud_migrations_config_json_source

You add the machine name of your module to dependencies > enforced > module array. This adds an enforced dependency on your own module. The effect is that the migration will be removed from Drupal’s active configuration when your custom module is uninstalled. Note that the top level dependencies array can have others keys in addition to enforced. For example: config and module. Learning more about them is left as an exercise for the curious reader.

It is important not to confuse the dependencies and migration_dependencies options. The former is provided by Drupal’s configuration management system and was just explained. The latter is provided by the Migrate API and is used to declare migrations that need be imported in advance. Read this article to know more about this feature. The following snippet shows an example:

uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
  enforced:
    module:
      - ud_migrations_config_json_source
migration_dependencies:
  required:
    - udm_config_json_source_image
    - udm_config_json_source_paragraph
  optional: []

What did you learn in today’s blog post? Did you know that you can manage migrations in two ways: code or configuration? Did you know that file name and location as well as workflows need to be adjusted depending on which approach you follow? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Workflows and benefits of managing Drupal migrations as configuration entities

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.