Welcome to the Drutopia Platform by Agaric, a cooperatively hosted website builder for grassroots groups.

Collaboratively building the future of online support for on-the-ground organizing.

Everything needed to grow a movement— together.  Simple, easy website building tools for big, complex work.

Drutopia is a flexible content management system with many features built specially for grassroots organizations.  It already helps groups share goals, call for action, collect donations, and report progress.  Most important, Drutopia's developers seek to design ongoing improvements and capabilities with groups organizing for a better world.

The Drutopia Platform by Agaric combines the ease of use you find from software as a service website builders (like Wix and Squarespace) with the freedom and control of free (libre) software.  It is LibreSaaS like Ghost(Pro) or WordPress.com, but it is built for and with grassroots groups.  Importantly, the platform is collectively controlled by the people who rely on it.

Please let us know what would be useful toward your organizing efforts.  We are looking forward to learning more about you and your goals and building this platform with you!

Surviving a civil war has a way of making a person focus on the present. Putting in a solid day of work and taking time with family tend to come before putting items on an agenda for a board meeting, even if one owns the company.

Most of the 30 workers at Brookline-based A Yard and a Half Landscaping, who have chosen to take ownership of their company and transform it into a cooperative, are immigrants from El Salvador. From 1980 to 1992, El Salvador endured a civil war in which 80,000 people died—more than two percent of the population—including many, many noncombatants, most killed by the US- and oligarch-backed anti-Communist military government. The violence still echoes.

The Yard and a Half landscapers are laying a new path of owning and managing the company themselves. To do so successfully, they must cultivate a culture of involvement in decision-making. Carolyn Edsell-Vetter is the first CEO and President of A Yard and a Half Landscaping Cooperative after its transition from ownership by its founder. Carolyn, who was the featured speaker at the Boston worker coop meetup on November 17, is determined to work herself out of presidency of the board. Along with ownership, all worker-owners must take on more leadership roles, including board officerships (all worker-owners are already on the board of directors).

For many at the meetup, the challenges Carolyn recounted about making a cooperative work matched experiences of our own, even though none of the rest of us were part of a transition from a non-coop company. Indeed, those of us in and out of cooperatives found much to discuss, from democracy to dollars.

How we talk about work and other things in worker-owned cooperatives was itself a topic of interest. Part of this is treating everyone with respect and trying to remove oppressive language, words and phrases or statements that intentionally or unintentionally reinforce racism, sexism, classism, ableism, and gender conformity.

"We spend about 30 minutes a working day working on our language," said James Bachez of Boston Collective Delivery, "and not the lunch hour." Collective organization calls for a "whole new language just beginning to understand." James stood up and paced while he thought and spoke. "Capitalist businesses talk about culture all the time," as something set from the top. Company culture is key for cooperatives too, but it comes from everyone. "The coop develops its culture. It's there. Whatever bubbles up to the top. I feel far more effective as part of a cooperative."

Most agreed that working on communication needs to be a constant, conscious effort. Collaboration, like other skills, should not be left to chance. Carolyn's praise of the AORTA Collective for its help with cooperative assistance was echoed by other cooperatives in attendance. Carolyn further thanked the Boston Interpreters Collective for making multilingual workshops possible.

The business of co-owning a business calls for financial literacy for all workers, too. "Everyone needs to know how to read a P&L", said Carolyn, referring to a statement that shows the profit and loss of a business. Used to see and estimate sales and expenses, it is fundamental to understanding any business's financial health.

We talked a great deal about fair compensation and balancing effort and types of work. A Yard and A Half does not have the same pay rate for all workers, and exactly what to pay everyone "has been really contentious. We inherited the wage structure from the previous owner," said Carolyn. They wanted their starting wages to be higher, but now people coming in can get almost as much as a crew leader who has been working for ten years. Either all pay rates have to bump up proportionately, or pay differentials have to flatten and everyone needs to be all right with that. "Thirty-five percent of cost to field labor would make us uncompetitive."

Like Carolyn, most felt there could be fair tiered systems in cooperatives, but no one felt they had one permanent answer. "This is how worker cooperatives reconcile the valuation of our labor in a capitalist system," said James, "while building something new." Most felt equal pay to be a good goal, but also that more experience, more responsibility, or just doing boring work no one else wanted to do could all justify more money. Sharing skills and sharing different kinds of work might be ideal, but isn't always practical and different work can be treated differently to have an overall just outcome. One attendee said she learned in Argentina that retirement age for physical labor is set lower, at 50 years.

Whatever the pay, there's no replacement for enjoying what one does. "Landscaping, you just have to love it," Carolyn said, "to make it through one season. We've had people come to [start their first day of] work and turn in their work clothes at lunch break." For those who like all that physical work, changing landscapes, making things beautiful, are extremely worthwhile.

A final big topic we talked about was hierarchy. A cooperative does not mean you don't have one person in charge of certain domains. Hierarchies help accomplish specific tasks efficiently, but they need not be permanent, and the structure adopted for a specific purpose or project in a cooperative is "a consensual hierarchy".

As interesting as our common ground was, the transition-specific experiences were also fascinating. A Yard and a Half has some great advantages coming from a transition— an established, profitable business and the passed-on wisdom from their founder, including advice which Carolyn said they are just beginning to recognize the truth of: "You either have to grow or you'll lose ground." The clientele is largely friendly to the progressive business that was built and the worker cooperative that it has become: hippies done good, people who want their kids or grandkids to be able to run around on the grass and eat dirt without being poisoned, thanks to a decision to stop using chemical pesticides and adopt organic practices.

Unexpected challenges of A Yard and a Half's transition to a cooperative corporation included having to re-establish the company's credit. Some businesses wanted personal guarantees from all of the owners for debts. They had to discuss and ultimately accept that some worker-owners—with homes and credit cards—were putting more at risk than others.

Transitioning from a hierarchical structure or creating a new worker cooperative both mean learning to self-govern. It is work that all in attendance recognized as hard but affirmed as worthwhile. Worker cooperatives offer control over our own work, increased control over our lives, and the promise of, perhaps, an economic base that can support collective liberty and justice.

A Yard and a Half Landscaping was not the only inspiring worker-owned cooperative we got to learn about that night. Boston Collective Delivery is one of three cooperatives in a fledgling federation that plans to grow, and in five years or so apply to be accepted into Mondragon, the largest federation of worker-owned cooperatives in the world— but we hope to learn and write about more on that later. There's no reason for you to wait if you're anywhere near Boston though! Meetup coordinator Micky Metts, James, many other attendees, and many more look forward to meeting interested people at WORC'N's holiday party this evening at Industry Lab. Join the meetup group for notice of new events or the WORC'N discussion list for discussion of worker cooperatives in the Boston area, including notice of these meetups.

When writing a bio, think about the voice and tone you'd like to use and the audience you are speaking to. What information will site visitors want to know about this person? What questions are they trying to answer when they visit your team's profile pages?

If written in the first person, this is a great opportunity for someone's personality to come through.

Sign up to be notified when Agaric gives a migration training:

As starving authors we at Agaric don't have a lot of cash to burn right now, but we've thrown $25 in the project to make it possible to subscribe to drupal.org issues without commenting. (On top of whatever we donated when this request for funding went out a year and a half ago).

Drupal's official documentation for the Migrations and Migrate API
The guide to the UI is still just a stub: https://www.drupal.org/docs/8/migrating-to-drupal
The Migrate API documentation is pretty comprehensive, and includes ways to use it: https://www.drupal.org/docs/8/api/migrate-api/

Building a custom migration in Drupal 8 blog series by Tess Flynn (socketwench)
https://deninet.com/blog/1615/building-custom-migration-drupal-8-part-1-getting-started

Migrate All the Things presentation by Dave Vasilevsky
https://events.drupal.org/baltimore2017/sessions/migrate-all-things

Migrating Multilingual Content in Drupal 8 presentation by Jigar Mehta
https://www.youtube.com/watch?v=L3SQiIAlvhA
with slides.

Migrating SQL to Drupal 8 with Migrate Tools and Migrate Plus
https://colorfield.be/blog/migrating-sql-in-drupal-8-with-migrate-tools-and-migrate-plus

Avoid sending emails while doing a migration on Drupal 8 by David Valdez
http://agaric.com/blogs/avoid-sending-emails-while-doing-migration-drupal-8

Slides from Mauricio Dinarte and Benjamin Melançon's presentation at DrupalCamp Twin Cities 2017:
https://docs.google.com/presentation/d/1rgEirVassqCRjgDqJ7p32nYcRRVACdi3x9Te7rOVSLk/

Extra tips, not in the slide deck:

* Use --limit while working on a large migration.
* Use git! Commit all changes to the migrate_plus.migration*.yml files you're working on. Then you can export configuration (such as for changes made to your field definitions) without fear, and git checkout your migrate_plus.migration*.yml file or files, and then import it all.

Thanks for signing up to hear from Agaric on matters involving movements for justice and liberty!

In the previous posts we talked about option to manage migrations as configuration entities and some of the benefits this brings. Today, we are going to learn another useful feature provided by the Migrate Plus module: migration groups. We are going to see how they can be used to execute migrations together and share configuration among them. Let’s get started.

Understanding migration groups

The Migrate Plus module defines a new configuration entity called migration group. When the module is enabled, each migration can define one group they belong to. This serves two purposes:

1. It is possible to execute operations per group. For example, you can import or rollback all migrations in the same group with one Drush command provided by the Migrate Tools module.
2. It is is possible to declare shared configuration for all the migrations within a group. For example, if they use the same source file, the value can be set in a single place: the migration group.

To demonstrate how to leverage migration groups, we will convert the CSV source example to use migrations defined as configuration and groups. You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD configuration group migration (CSV source) whose machine name is ud_migrations_config_group_csv_source. It comes with three migrations: udm_config_group_csv_source_paragraph, udm_config_group_csv_source_image, and  udm_config_group_csv_source_node. Additionally, the demo module provides the udm_config_group_csv_source group.

Note: The Migrate Tools module provides a user interface for managing migrations defined as configuration. It is available under “Structure > Migrations” at /admin/structure/migrate. This user interface will be explained in a future entry. For today’s example, it is assumed that migrations are executed using the Drush commands provided by Migrate Plus. In the past we have used the Migrate Run module to execute migrations, but this module does not offer the ability to import or rollback migrations per group.

Creating a migration group

The migration groups are defined in YAML files using the following naming convention: migrate_plus.migration_group.[migration_group_id].yml. Because they are configuration entities, they need to be placed in the config/install directory of your module. Files placed in that directory following that pattern will be synced into Drupal’s active configuration when the module is installed for the first time (only). If you need to update the migration groups, you make the modifications to the files and then sync the configuration again. More details on this workflow can be found in this article. The following snippet shows an example migration group:

uuid: e88e28cc-94e4-4039-ae37-c1e3217fc0c4
id: udm_config_group_csv_source
label: 'UD Config Group (CSV source)'
description: 'A container for migrations about individuals and their favorite books. Learn more at https://understanddrupal.com/migrations.'
source_type: 'CSV resource'
shared_configuration: null

The uuid key is optional. If not set, the configuration management system will create one automatically and assign it to the migration group. Setting one simplifies the workflow for updating configuration entities as explained in this article. The id key is required. Its value is used to associate individual migrations to this particular group.

The label, description, and source_type keys are used to give details about the migration. Their value appear in the user interface provided by Migrate Tools. label is required and serves as the name of the group. description is optional and provides more information about the group. source_type is optional and gives context about the type of source you are migrating from. For example, "Drupal 7", "WordPress", "CSV file", etc.

To associate a migration to a group, set the migration_group key in the migration definition file: For example:

uuid: 97179435-ca90-434b-abe0-57188a73a0bf
id: udm_config_group_csv_source_node
label: 'UD configuration host node migration for migration group example (CSV source)'
migration_group: udm_config_group_csv_source
source: ...
process: ...
destination: ...
migration_dependencies: ...

Note that if you omit the migration_group key, it will default to a null value meaning the migration is not associated with any group. You will still be able to execute the migration from the command line, but it will not appear in the user interface provided by Migrate Tools. If you want the migration to be available in the user interface without creating a new group, you can set the migration_group key to default. This group is automatically created by Migrate Plus and can be used as a generic container for migrations.

Organizing and executing migrations

Migration groups are used to organize migrations. Migration projects usually involve several types of elements to import. For example, book reports, events, subscriptions, user accounts, etc. Each of them might require multiple migrations to be completed. Let’s consider a news articles migration. The "book report" content type has many entity reference fields: book cover (image), support documents (file), tags (taxonomy term), author (user), citations (paragraphs). In this case, you will have one primary node migration that depends on five migrations of multiple types. You can put all of them in the same group and execute them together.

It is very important not to confuse migration groups with migration dependencies. In the previous example, the primary book report node migration should still list all its dependencies in the migration_dependencies section of its definition file. Otherwise, there is no guarantee that the five migrations it depends on will be executed in advance. This could cause problems if the primary node migration is executed before images, files, taxonomy terms, users, or paragraphs have already been imported into the system.

It is possible to execute all migrations in a group by issuing a single Drush with the --group flag. This is supported by the import and rollback commands exposed by Migrate Tools. For example, drush migrate:import --group='udm_config_group_csv_source'. Note that as of this writing, there is no way to run all migrations in a group in a single operation from the user interface. You could import the main migration and the system will make sure that if any explicit dependency is set, they will be run in advance. If the group contained more migrations than the ones listed as dependencies, those will not be imported. Moreover, migration dependencies are only executed automatically for import operations. Dependent migrations will not be rolled back automatically if the main migration is rolled back individually.

Note: This example assumes you are using Drush to execute the migrations. At the time of this writing, it is not possible to rollback a CSV migration from the user interface. See this issue in the Migrate Source CSV for more context.

Sharing configuration with migration groups

Arguably, the major benefit of migration groups is the ability to share configuration among migrations. In the example, there are three migrations all reading from CSV files. Some configurations like the source plugin and header_offset can be shared. The following snippet shows an example of declaring shared configuration in the migration group for the CSV example:

uuid: e88e28cc-94e4-4039-ae37-c1e3217fc0c4
id: udm_config_group_csv_source
label: 'UD Config Group (CSV source)'
description: 'A container for migrations about individuals and their favorite books. Learn more at https://understanddrupal.com/migrations.'
source_type: 'CSV resource'
shared_configuration:
  dependencies:
    enforced:
      module:
        - ud_migrations_config_group_csv_source
  migration_tags:
    - UD Config Group (CSV Source)
    - UD Example
  source:
    plugin: csv
    # It is assumed that CSV files do not contain a headers row. This can be
    # overridden for migrations where that is not the case.
    header_offset: null

Any configuration that can be set in a regular migration definition file can be set under the shared_configuration key. When the migrate system loads the migration, it will read the migration group it belongs to, and pull any shared configuration that is defined. If both the migration and the group provide a value for the same key, the one defined in the migration definition file will override the one set in the migration group. If a key only exists in the group, it will be added to the migration when the definition file is loaded.

In the example, dependencies, migration_tag, and source options are being set. They will apply to all migrations that belong to the udm_config_group_csv_source group. If you updated any of these values, the changes would propagate to every migration in the group. Remember that you would need to sync the migration group configuration for the update to take effect. You can do that with partial configuration imports as explained in this article.

Any configuration set in the group can be overridden in specific migrations. In the example, the header_offset is set to null which means the CSV files do not contain a header row. The node migration uses a CSV file that contains a header row so that configuration needs to be redeclared. The following snippet shows how to do it:

uuid: 97179435-ca90-434b-abe0-57188a73a0bf
id: udm_config_group_csv_source_node
label: 'UD configuration host node migration for migration group example (CSV source)'
# Any configuration defined in the migration group can be overridden here
# by re-declaring the configuration and assigning a value.
# dependencies inherited from migration group.
# migration_tags inherited from migration group.
migration_group: udm_config_group_csv_source
source:
  # plugin inherited from migration group.
  path: modules/custom/ud_migrations/ud_migrations_csv_source/sources/udm_people.csv
  ids: [unique_id]
  # This overrides the header_offset defined in the group. The referenced CSV
  # file includes headers in the first row. Thus, a value of 0 is used.
  header_offset: 0
process: ...
destination: ...
migration_dependencies: ...

Another example would be multiple migrations reading from a remote JSON. Let’s say that instead of fetching a remote file, you want to read a local file. The only file you would have to update is the migration group. Change the data_fetcher_plugin key to file and the urls array to the path to the local file. You can try this with the ud_migrations_config_group_json_source module from the demo repository.

What did you learn in today’s blog post? Did the know that migration groups can be used to share configuration among different migrations? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: What is the difference between migration tags and migration groups in Drupal?

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.