Sign up to be notified when Agaric gives a migration training:
Election-first organizing is failing us
We need to move to movement-first organizing
Welcome to the Drutopia Platform by Agaric, a cooperatively hosted website builder for grassroots groups.
Collaboratively building the future of online support for on-the-ground organizing.
Everything needed to grow a movement— together. Simple, easy website building tools for big, complex work.
Drutopia is a flexible content management system with many features built specially for grassroots organizations. It already helps groups share goals, call for action, collect donations, and report progress. Most important, Drutopia's developers seek to design ongoing improvements and capabilities with groups organizing for a better world.
The Drutopia Platform by Agaric combines the ease of use you find from software as a service website builders (like Wix and Squarespace) with the freedom and control of free (libre) software. It is LibreSaaS like Ghost(Pro) or WordPress.com, but it is built for and with grassroots groups. Importantly, the platform is collectively controlled by the people who rely on it.
Please let us know what would be useful toward your organizing efforts. We are looking forward to learning more about you and your goals and building this platform with you!
Each month we host an event at Industry Lab in Cambridge, Massachusetts to discuss cooperative work and culture. Our events are usually an informal pot luck gathering.
We meet to help one another explore, form, join, or improve businesses without bosses. Members of worker cooperatives and collectives frequently share our experiences, our challenges, and the processes that are working for us. Sometimes we have guest speakers from successful cooperatives who share stories of how they started their business and how they organize and structure the business. The meetup also engages people that are not yet a part of a worker owned cooperative, but have interest and questions.
At our first meetup we were lucky to have Gonzalo Chomón of Evovelo, a company creating solar powered vehicles, visiting from Spain. He gave detailed information on cooperative law and the cooperative movement in his country. Spain is the home of Mondragon, the world's largest cooperative organization.
Another evening gathering featured A Yard & A Half Landscaping, a successful local cooperative of more than 30 members created by a transition from a single-owner business. Member Carolyn Edsell-Vetter related A Yard & A Half's impressive history and current challenges as its worker-owners grow the cooperative.
At the end of 2014 we were honored to host a holiday party for WORC'N. For our first meetup of 2015, Yochai Gal and Matt Gabrenya from the Boston TechCollective gave an in-depth look into their cooperative beginnings (a still-going San Francisco TechCollective) and how they have attained excellence in the computer repair business. People were actively engaged in a discussion about the roles and duties that exist in a thriving cooperative, and how we can best go about integrating new members.
Our meetups are held at the Industry Lab at 288 Norfolk Street in Cambridge, Massachusetts. The venue is an eclectic environment with a mix of product based companies, inventors, artists and us, the only web developers to date. The Boston/Cambridge Worker Cooperative group on meetup.com has 60 members after just a few months of existence, and each gathering usually has at least a dozen people coming. We invite you to join us at a meetup: sign up at the Meetup group or contact us to be alerted to the next event!
Surviving a civil war has a way of making a person focus on the present. Putting in a solid day of work and taking time with family tend to come before putting items on an agenda for a board meeting, even if one owns the company.
Most of the 30 workers at Brookline-based A Yard and a Half Landscaping, who have chosen to take ownership of their company and transform it into a cooperative, are immigrants from El Salvador. From 1980 to 1992, El Salvador endured a civil war in which 80,000 people died—more than two percent of the population—including many, many noncombatants, most killed by the US- and oligarch-backed anti-Communist military government. The violence still echoes.
The Yard and a Half landscapers are laying a new path of owning and managing the company themselves. To do so successfully, they must cultivate a culture of involvement in decision-making. Carolyn Edsell-Vetter is the first CEO and President of A Yard and a Half Landscaping Cooperative after its transition from ownership by its founder. Carolyn, who was the featured speaker at the Boston worker coop meetup on November 17, is determined to work herself out of presidency of the board. Along with ownership, all worker-owners must take on more leadership roles, including board officerships (all worker-owners are already on the board of directors).
For many at the meetup, the challenges Carolyn recounted about making a cooperative work matched experiences of our own, even though none of the rest of us were part of a transition from a non-coop company. Indeed, those of us in and out of cooperatives found much to discuss, from democracy to dollars.
How we talk about work and other things in worker-owned cooperatives was itself a topic of interest. Part of this is treating everyone with respect and trying to remove oppressive language, words and phrases or statements that intentionally or unintentionally reinforce racism, sexism, classism, ableism, and gender conformity.
"We spend about 30 minutes a working day working on our language," said James Bachez of Boston Collective Delivery, "and not the lunch hour." Collective organization calls for a "whole new language just beginning to understand." James stood up and paced while he thought and spoke. "Capitalist businesses talk about culture all the time," as something set from the top. Company culture is key for cooperatives too, but it comes from everyone. "The coop develops its culture. It's there. Whatever bubbles up to the top. I feel far more effective as part of a cooperative."
Most agreed that working on communication needs to be a constant, conscious effort. Collaboration, like other skills, should not be left to chance. Carolyn's praise of the
The business of co-owning a business calls for financial literacy for all workers, too. "Everyone needs to know how to read a P&L", said Carolyn, referring to a statement that shows the profit and loss of a business. Used to see and estimate sales and expenses, it is fundamental to understanding any business's financial health.
We talked a great deal about fair compensation and balancing effort and types of work. A Yard and A Half does not have the same pay rate for all workers, and exactly what to pay everyone "has been really contentious. We inherited the wage structure from the previous owner," said Carolyn. They wanted their starting wages to be higher, but now people coming in can get almost as much as a crew leader who has been working for ten years. Either all pay rates have to bump up proportionately, or pay differentials have to flatten and everyone needs to be all right with that. "Thirty-five percent of cost to field labor would make us uncompetitive."
Like Carolyn, most felt there could be fair tiered systems in cooperatives, but no one felt they had one permanent answer. "This is how worker cooperatives reconcile the valuation of our labor in a capitalist system," said James, "while building something new." Most felt equal pay to be a good goal, but also that more experience, more responsibility, or just doing boring work no one else wanted to do could all justify more money. Sharing skills and sharing different kinds of work might be ideal, but isn't always practical and different work can be treated differently to have an overall just outcome. One attendee said she learned in Argentina that retirement age for physical labor is set lower, at 50 years.
Whatever the pay, there's no replacement for enjoying what one does. "Landscaping, you just have to love it," Carolyn said, "to make it through one season. We've had people come to [start their first day of] work and turn in their work clothes at lunch break." For those who like all that physical work, changing landscapes, making things beautiful, are extremely worthwhile.
A final big topic we talked about was hierarchy. A cooperative does not mean you don't have one person in charge of certain domains. Hierarchies help accomplish specific tasks efficiently, but they need not be permanent, and the structure adopted for a specific purpose or project in a cooperative is "a consensual hierarchy".
As interesting as our common ground was, the transition-specific experiences were also fascinating. A Yard and a Half has some great advantages coming from a transition— an established, profitable business and the passed-on wisdom from their founder, including advice which Carolyn said they are just beginning to recognize the truth of: "You either have to grow or you'll lose ground." The clientele is largely friendly to the progressive business that was built and the worker cooperative that it has become: hippies done good, people who want their kids or grandkids to be able to run around on the grass and eat dirt without being poisoned, thanks to a decision to stop using chemical pesticides and adopt organic practices.
Unexpected challenges of A Yard and a Half's transition to a cooperative corporation included having to re-establish the company's credit. Some businesses wanted personal guarantees from all of the owners for debts. They had to discuss and ultimately accept that some worker-owners—with homes and credit cards—were putting more at risk than others.
Transitioning from a hierarchical structure or creating a new worker cooperative both mean learning to self-govern. It is work that all in attendance recognized as hard but affirmed as worthwhile. Worker cooperatives offer control over our own work, increased control over our lives, and the promise of, perhaps, an economic base that can support collective liberty and justice.
A Yard and a Half Landscaping was not the only inspiring worker-owned cooperative we got to learn about that night. Boston Collective Delivery is one of three cooperatives in a fledgling federation that plans to grow, and in five years or so apply to be accepted into Mondragon, the largest federation of worker-owned cooperatives in the world— but we hope to learn and write about more on that later. There's no reason for you to wait if you're anywhere near Boston though! Meetup coordinator Micky Metts, James, many other attendees, and many more look forward to meeting interested people at WORC'N's holiday party this evening at Industry Lab. Join the meetup group for notice of new events or the WORC'N discussion list for discussion of worker cooperatives in the Boston area, including notice of these meetups.
When continuing development of a web site, big changes occur every so often. One such change that may occur, frequently as a result of another change, is a bulk update of URLs. When this is necessary, you can greatly improve the response time experienced by your users—as they are redirected from the old path to the new path—by using a handy directive offered in Apache's mod_rewrite called RewriteMap.
At Agaric we regularly turn to Drupal for it's power and flexibility, so one might question why we didn't leverage Drupal's support for handling redirects. When we see an opportunity for our software/system to respond "as early as it can", it is worth investigating how that is handled. Apache handles redirects itself, making it entirely unnecessary to hand-off to PHP, never mind Drupal bootstrapping and retrieving up a redirect record in a database, just to tell a browser (or a search engine) to look somewhere else.
There were two conditions that existed making use of RewriteMap a great candidate. For one, there will be no changes to the list of redirects once they are set: these are for historical purposes only (the old URLs are no longer exposed anywhere else on the site). Also, because we could make the full set of hundreds of redirects via a single RewriteRule—thanks to the substitution capability afforded by RewriteMap—this solution offered a fitting and concise solution.
So, what did we do, and how did we do it?
We started with an existing set of URLs that followed the pattern: http://example.com/user-info/ID[/tab-name]. Subsequently we implemented a module on the site that produced aliases for our user page URLs. The new patten to the page was then (given exceptions for multiple J Smiths, etc via the suffix): http://example.com/user-info/firstname-lastname[-suffix#][/tab-name]. The mapping of ID to firstname-lastname[-suffix#] was readily available within Drupal, so we used an update_hook to write out the existing mappings to a file (in the Drupal public files folder, since we know that's writable by Drupal) . This file (which I called 'staffmapping.txt') is what we used for a simple text-based rewrite map. Sample output of the update hook looked like this:
# User ID to Name mapping: 1 admin-admin 2 john-smith 3 john-smith-2 4 jane-smith
The format of this file is pretty straight-forward: comments can be started on any line with a #, and the mapping lines themselves are composed of {lookupValue}{whitespace}{replacementValue}.
To actually consume this mapping somewhere in our rules, we must let Apache know about the mapping file itself. This is done with a RewriteMap directive, which can be placed in the Server config or else inside a VirtualHost directive. The format of the RewriteMap looks like this: RewriteMap MapName MapType:MapSource. In our case, the file is a simple text file mapping, so the MapType is 'txt'. The resulting string added to our VirtualHost section is then: RewriteMap staffremap txt:/path/to/staffmapping.txt This directive makes this rewrite mapping file available under the name "staffremap" in our RewriteRules. There are other MapTypes, including ones that uses random selection for the replacement values from a text file, using a hash map rather than a text file, using an internal function, or even using an external program or script to generate replacement values.
Now it's time to actually change incoming URLs using this mapping file, providing the 301 redirect we need. The rewrite rule we used, looks like this:
RewriteRule ^user-detail/([0-9]+)(.*) /user-detail/${staffremap:$1}$2 [R=301,L]The initial argument to the rewrite rule identifies what incoming URLs this rule applies to. This is the string: "^user-detail/([0-9]+)(.*)". This particular rule looks for URLs starting with (signified by the special character ^) the string "user-detail/", then followed by one or more numbers: ([0-9]+), and finally, anything else that might appear at the end of the string: "(.*)". There's a particular feature of regex being used here as well: each of search terms in parenthesis are captured (or tagged) by the regex processor which then provides some references that can be used in the replacement string portion. These are available with $<captured position>—so, the first value captured by parenthesis is available in "$1"—this would be the user ID, and the second in "$2"—which for this expression would be anything else appearing after the user ID.
Following the whitespace is our new target URL expression: "/user-detail/${staffremap:$1}$2". We're keeping the beginning of the URL the same, and then following the expression syntax "${rewritemap:lookupvalue}", which in our case is: "${staffremap:$1}" we find the new user-name URL. This section could be read as: take the value from the rewrite map called "staffremap", where the lookup value is $1 (the first tagged expression in the search: the numeric value) and return the substitution value from that map in place of this expression. So, if we were attempting to visit the old URL /user-detail/1/about, the staffremap provides the value "admin-admin" from our table. The final portion of the replacement URL (which is just $2) copies everything else that was passed on the URL through to the redirected URL. So, for example, /user-detail/1/about includes the /about portion of the URL in the ultimate redirect URL: /user-detail/admin-admin/about
The final section of the sample RewriteRule is for applying additional flags. In this case, we are specifying the response status of 301, and the L indicates to mod_rewrite that this is the last rule it should process.
That's basically it! We've gone from an old URL pattern, to a new one with a redirect mapping file, and only two directives. For an added performance perk, especially if your list of lookup and replacement values is rather lengthy, you can easily change your text table file (type txt) with a HashMap (type dbm) that Apache's mod_rewrite also understands using a quick command and directive adjustment. Following our example, we'll first run:
$> httxt2dbm -i staffrepam.txt -o staffremap.map
Now that we have a hashmap file, we can adjust our RewriteMap directive accordingly, changing the type to map, and of course updating the file name, which becomes:
RewriteMap staffremap dbm:/path/to/staffremap.map
RewriteMap substitutions provide a straight-forward, and high-performance method for pretty extensive enhancement of RewriteRules. If you are not familiar with RewriteRules generally, at some point you should consider reviewing the Apache documentation on mod_rewrite—it's worthwhile knowledge to have.
The entity_generate process plugin receives a value and checks if there is an entity with that name and if the term exists then uses it and if it does not then creates it (which is precisely what I need).
So, here is a snippet of the article migration YAML file using the entity_generate plugin:
id: blog migration_group: Drupal label: Blog source: plugin: d7_node node_type: blog destination: plugin: entity:node process: status: status created: created field_tags: plugin: sub_process source: field_tags process: target_id: - plugin: entity_generate source: name value_key: name bundle_key: vid bundle: tags entity_type: taxonomy_term ignore_case: true …
In our field_tags field we are using the Drupal 7 field_tags Values We are going to read the entities and pass that value into the entity_generate plugin to create the entities. In this example, there is a problem. The d7_node migrate plugin (included in the migrate module) provides the taxonomy term IDs and this will make the entity_generate plugin create some taxonomy terms using the IDs as the term names, and this is not what we want.
So what I need to do is to get from somewhere the terms' names, not their ids. To do that I need to add an extra `source property`.
First, we need to create a new custom module and in there create a source plugin which extends the Node process plugin, something like this (let's say that our custom module’s name is my_migration):
Create the file:
my_migration/src/Plugin/migrate/source/MyNode.php
And the Content of MyNode file should have this code:
namespace Drupal\my_migration\Plugin\migrate\source; use Drupal\migrate\Row; use Drupal\node\Plugin\migrate\source\d7\Node; /** * Adds a source property with the taxonomy term names. * * @MigrateSource( * id = “my_node", * source_module = "node" * ) */ class MyNode extends Node { public function prepareRow(Row $row) { $nid = $row->getSourceProperty('nid'); // Get the taxonomy tags names. $tags = $this->getFieldValues('node', 'field_tags', $nid); $names = []; foreach ($tags as $tag) { $tids[] = $tag['tid']; } if (!$tids) { $names = []; } else { $query = $this->select('taxonomy_term_data', 't'); $query->condition('tid', $tids, 'IN'); $query->addField('t', 'name'); $result = $query->execute()->fetchCol(); $names[] = ['name' => $result['name']]; foreach ($result as $term_name) { $names[] = ['name' => $term_name]; } } $row->setSourceProperty('field_tags_names', $names); return parent::prepareRow($row); } }
The most important part of this code is:
$nid = $row->getSourceProperty('nid'); // Get the taxonomy tags names. $tags = $this->getFieldValues('node', 'field_tags', $nid); $names = []; foreach ($tags as $tag) { $tid = $tag['tid']; $query = $this->select('taxonomy_term_data', 't'); $query->condition('tid', $tid); $query->addField('t', 'name'); $result = $query->execute()->fetchAssoc(); $names[] = ['name' => $result['name']]; } $row->setSourceProperty('field_tags_names', $names);
It does the following things:
- It reads the nid of the article
- Gets all the terms' IDs of the field_tags field
- For each ID, it gets the name of the term and puts it in the `names` array
- Finally, it sets this value inside the $row using the `setsourceProperty` method.
Now our rows will have a property called fields_tags_names with the terms' names, and we can pass this data to the entity_generate plugin.
We need to make a few adjustments in our initial migration file. First and most important, update the source plugin to use our new source plugin:
source: plugin: my_node …
And update the source in the `field_tags` field to use the new `field_tags_names` source property.
… field_tags: plugin: sub_process source: field_tags_names ….
The final migration file looks like this:
id: blog migration_group: Drupal label: Blog source: plugin: my_node node_type: blog destination: plugin: entity:node process: status: status created: created field_tags: plugin: sub_process source: field_tags_names process: target_id: - plugin: entity_generate source: name value_key: name bundle_key: vid bundle: tags entity_type: taxonomy_term ignore_case: true …
And that’s it; if we run the migration, it will create on the fly the terms that do not exist and use them if they do exist.
Get the most out of (and into) your page cache: Leave AJAX disabled in your Views, especially with exposed filters
Enabling AJAX for a Views page can have a performance-harming side effect one might not think of. On a recently built site we observed a relatively low Varnish cache hit rate of 30% using the Munin plugin for Varnish. This hit rate was much lower than expected after prelaunch tests with Jmeter. (The site has almost exclusively anonymous visitors and if caching the pages worked efficiently the same low cost server could handle a lot more traffic.)
An analysis of the most visited paths on the live site showed one ranking two orders of magnitude above all else: views/ajax.
The Views pages on Studio Daniel Libeskind have exposed filters, and with AJAX enabled new data is fetched via POST request to views/ajax when a visitor applies a filter. This happens because the Drupal AJAX Framework leveraged by views exclusively uses POST requests. See issue Ensure it is possible to use AJAX with GET requests for a good explanation of why it is currently this way and the effort to allow GET requests in later versions of Drupal.
As a consequence of all AJAX views using the same path and all filter information being hidden in the request body, Varnish has no straightforward means of caching the content coming from views/ajax. Another downside: It's not easy to share a link to a filtered version of such a page.
If AJAX is not enabled (which is the default) filters are implemented as query parameters in the URL so there's a unique URL for each filtered page. That plays well with reverse proxy caches like Varnish and works well for people wanting to share links, so we disabled AJAX again and the Varnish page cache hit rate has risen to over 90% since.
The International Summit of Cooperatives convened in Quebec in 2016. The general message of the conference was that cooperatives are everywhere and one only needs to raise awareness for this idea to spread. That seems to be happening as evidenced by the attendance at this conference - 3000 people from 117+ countries according to the International Co-operative Alliance (ICA.coop) one of the sponsors of #ISCOOP2016.
I have been to many cooperative conferences and events, but this one was very different. From the facility to the attendees, the event had an air of style and conform that went beyond attire. I was swimming in a sea of 70-80% middle-aged men in black suits as far as the eye could see. There were also a few groups I saw that were wearing indigenous dress from India, Nepal, Chile, and Congo. Quebec is a mixture of modern and old culture. There were women in authentic Breton garb serving food in the restaurant we visited for lunch, in Old Quebec City, but they were not represented at the coop summit.
The largest sponsors of the Summit were the Canadian Government, Canada Economic Development, ICA International Co-operative Alliance and DesJardins. You can see a full listing of all the cooperative sponsors for the event. The attendees were mostly members of Agriculture and Financial coops, both small and large. When I say large, I am talking thousands of members. The point was brought up and highlighted that the International Co-operative Alliance represents close to one billion individual members. Statistics are calculated using the Alliance's formula based on active subscriptions. The ICA maintains the internationally recognised definition of a co-operative in the Statement on the Co-operative Identity and they represent 272 co-operative federations and organizations in 94 countries as of January 2014. The National Cooperative Bank released its annual report in 2015, listing the nation’s top 100 revenue-earning cooperative businesses. These 100 businesses posted revenue of approximately $243.2 billion.
A dominant presence by DesJardins, with over 6 million members, and the Boston Consulting Group (BCG). In my opinion, the BCG message was depressing and the same old Capitalistic message cloaked in a message of "growth" suggesting that "you must grow in order to prosper!" Luckily some cooperative panelists responded with how irresponsible it is to grow for growth sake. A member of DesJardins told me that he personally thinks the coop has gotten too big and we had a great talk about how empathy training should be available to larger entities. Marc Thomas, a DesJardins member, talked about helping people and how the DesJardins cooperative has made some positive changes for him and his community. They are much more willing to lend to smaller cooperatives and they have a host of connections to nurture a small business in start up phases. They play a role similar to a credit union on a much larger scale and they also have deep ties in the community and a network of cooperatives to connect new ideas to funding opportunities.
According to Howard Brodsky there are 50 cooperatives larger than Facebook. Brodsky is Co-Founder, Chairman, and Chief Executive Officer of CCA Global Partner. He is responsible for creating a cooperative retail powerhouse in the marketplace.
His message was most similar to all the coop conferences I had attended, yet he was much more vocal about how expansive the coop movement is - we are large and we are everywhere. In my opinion, Howard gets it and understands how empathy and caring figure into the movement. He touched on how this simply will not work if we are not honest and caring in our work. He spoke about how important "Stories" are and how they create bonds. The International Co-op Alliance (ica.coop) has built a wonderful way to share our narratives in this digital era - http://stories.coop
Trebor Scholz, a professor at the New School in NYC, and Nathan Schneider, a professor at University of Colorado Boulder, were each on a panel. Nathan's panel was on Multi Sector Activity and he talked about Platform Cooperativism as a way to bring cooperative communities together and how important it is to own the utilities and services we depend on. Trebor on the next panel framed platform coop as a movement and points to the recently published book "Ours to Hack and to Own" as the handbook to get involved in the movement. Copies of book are available from OR Books.
Both panels were lively and got a good response of people talking amongst each other after they ended. Attendees I talked to during the conference were diverse. People from Kenya and the Congo seemed to be the only ones shocked at the implications when I told them about free software. They had never heard of it. People from India that I met either said they knew about it or that they used it in their work. Other people from afar seemed bored and made excuses to not hear about it.
Translation for the speakers and panelists was stellar. No time lag at all and the team was professional and consistent. This did not carry over into the main event participants and attendees. The language barriers seemed to keep people from mingling outside of their party of friends. They sat in groups at the meals and at the sessions. The lunch/dinner seating was round table, with place settings ala extra forks etc. very convenient for conversations. Meals were also used as a venue for a sort of Keynote delivery that happened on several giant screens while the appetizers were served.
The event was full of Pros and positive energy, there were only a few minor Cons:
1. A lot of people in the crowd were unaware of free software, and almost no one used encryption.
2. Most panels were all male - even ones discussing diversity.
3. Some financial coops place most emphasis on growing, as if growth is the only measure of success and value.
Robert Reich got a standing ovation for his keynote with a message that coops are an important part of the business landscape. He spent most of his talk telling an anecdotal touchy feely story with the point that we all need to get along. He seems to be a progressive at times, but he still operates on a lesser of two evils mentality in a two party system - I ask, why won't he support a third party if he is in agreement with most of their platforms?
I spoke to many people throughout the three day event about what Agaric is and what we do. I also talked about Free Software and how it impacts cooperatives and their goals. Some were not aware of free software and the vital role it will play in the future success of cooperatives maintaining autonomy and privacy. I also spoke about Platform Cooperativism and Drutopia as a platform concept. Anyone seeking more information can sign up at the Drutopia.org website to be invited to discussions and have a voice in the process of building a platform cooperative from the ground up.
So, things are looking bright for cooperatives in the future. The cooperative branding and marketing needs building, and the network needs to keep expanding and cross-pollinating. The tireless work and dedication of small groups like the ICA is what makes this all happen. Coops do not need to be large, they need to be nimble and they need to be flexible. With apps like BuyCott it will be much easier to purchase responsibly, buy from cooperatives and support ethical companies. I just got the app and am happily surprised to find out how many dedicated cooperative people there are in the world shopping responsibly already! We can each do our part to make the network stronger and to bring the cooperative movements closer together. How would you bring something cooperative to your community? Even a small local event at your neighborhood coffee shop or a blog post or a Tweet can do a lot to raise the level of awareness of how strong we are together!
The results are in:
Typically an awesome event will end and as time passes, there will be little to no follow-up or tangible results that are published. You wonder if that great project you heard about is flourishing or forgotten. You can see the results of the workshops in Quebec in 2016 and rejoice in the knowledge that we are on our way to autonomy.
Our friend Chuck Bordman has written an excellent blog covering this conference here: Coopmatters.com
Today, we are going to talk about how to manage migrations as configuration entities. This functionality is provided by the Migrate Plus module. First, we will explain the difference between managing migrations as code or configuration. Then, we will show how to convert existing migrations. Finally, we will talk about some important options to include in migration configuration entities. Let’s get started.
Drupal migrations: code or configuration?
So far, we have been managing migrations as code. This is functionality provided out of the box. You write the migration definition file in YAML format. Then, you place it in the migrations directory of your module. If you need to update the migration, you make the modifications to the files and then rebuild caches. More details on the workflow for migrations managed in code can be found in this article.
Migrate Plus offers an alternative to this approach. It allows you to manage migrations as configuration entities. You still use YAML files to write the migration definition files, but their location and workflow is different. They need to be placed in a config/install directory. If you need to update the migration, you make the modifications to the files and then sync the configuration again. More details on this workflow can be found in this article.
There is one thing worth emphasizing. When managing migrations as code you need access to the file system to update and deploy the changes to the file. This is usually done by developers. When managing migrations as configuration, you can make updates via the user interface as long as you have permissions to sync the site’s configuration. This is usually done by site administrators. You might still have to modify files depending on how you manage your configuration. But the point is that file system access to update migrations is optional. Although not recommended, you can write, modify, and execute the migrations entirely via the user interface.
Transitioning to configuration entities
To demonstrate how to transition from code to configuration entities, we are going to convert the JSON migration example. You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD config JSON source migration whose machine name is udm_config_json_source. It comes with four migrations: udm_config_json_source_paragraph, udm_config_json_source_image, udm_config_json_source_node_local, and udm_config_json_source_node_remote.
The transition to configuration entities is a two step process. First, move the migration definition files from the migrations folder to a config/install folder. Second, rename the files so that they follow this pattern: migrate_plus.migration.[migration_id].yml. For example: migrate_plus.migration.udm_config_json_source_node_local.yml. And that’s it! Files placed in that directory following that pattern will be synced into Drupal’s active configuration when the module is installed for the first time (only). Note that changes to the files require a new synchronization operation for changes to take effect. Changing the files and rebuilding caches does not update the configuration as it was the case with migrations managed in code.
If you have the Migrate Plus module enabled, it will detect the migrations and you will be able to execute them. You can continue using the Drush commands provided the Migrate Run module. Alternatively, you can install the Migrate Tools module which provides Drush commands for running both types of migrations: code and configuration. Migrate Tools also offers a user interface for executing migrations. This user interface is only for migrations defined as configuration though. It is available at /admin/structure/migrate. For now, you can run the migrations using the following Drush command: drush migrate:import udm_config_json_source_node_local --execute-dependencies.
Note: For executing migrations in the command line, choose between Migrate Run or Migrate Tools. You pick one or the other, but not both as the commands provided by the two modules have the same name. Another thing to note is that the example uses Drush 9. There were major refactorings between versions 8 and 9 which included changes to the name of the commands.
UUIDs for migration configuration entities
When managing migrations as configuration, you can set extra options. Some are exposed by Migrate Plus while others come from Drupal’s configuration management system. Let’s see some examples.
The most important new option is defining a UUID for the migration definition file. This is optional, but adding one will greatly simplify the workflow to update migrations. The UUID is used to keep track of every piece of configuration in the system. When you add new configuration, Drupal will read the UUID value if provided and update that particular piece of configuration. Otherwise, it will create a UUID on the fly, attach it to the configuration definition, and then import it. That is why you want to set a UUID value manually. If changes need to be made, you want to update the same configuration, not create a new one. If no UUID was originally set, you can get the automatically created value by exporting the migration definition. The workflow for this is a bit complicated and error prone so always include a UUID with your migrations. This following snippet shows an example UUID:
uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'The UUID a string of 32 hexadecimal digits displayed in 5 groups. Each is separated by hyphens following this pattern: 8-4-4-4-12. In Drupal, two or more pieces of configuration cannot share the same value. Drupal will check the UUID and the type of configuration in sync operations. In this case the type is signaled by the migrate_plus.migration. prefix in the name of the migration definition file.
When using configuration entities, a single migration is identified by two different options. The uuid is used by the Drupal’s configuration system and the id is used by the Migrate API. Always make sure that this combination is kept the same when updating the files and syncing the configuration. Otherwise you might get hard to debug errors. Also, make sure you are importing the proper configuration type. The latter should not be something to worry about unless you utilize the user interface to export or import single configuration items.
If you do not have a UUID in advance for your migration, you can try one of these commands to generate it:
# Use Drupal's UUID service.
$ drush php:eval "echo \Drupal::service('uuid')->generate(). PHP_EOL;"
# Use a Drush command provided by the Devel module, if enabled.
$ drush devel:uuid
# Use a tool provided by your operating system, if available.
$ uuidgenAlternatively, you can search online for UUID v4 generators. There are many available.
Technical note: Drupal uses UUID v4 (RFC 4122 section 4.4) values which are generated by the `uuid` service. There is a separate class for validation purposes. Drupal might override the UUID service to use the most efficient generation method available. This could be using a PECL extension or a COM implementation for Windows.
Automatically deleting migration configuration entities
By default, configuration remains in the system even if the module that added it gets uninstalled. This can cause problems if your migration depends on custom migration plugins provided by your module. It is possible to enforce that migration entities get removed when your custom module is uninstalled. To do this, you leverage the dependencies option provided by Drupal’s configuration management system. The following snippet shows how to do it:
uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
enforced:
module:
- ud_migrations_config_json_sourceYou add the machine name of your module to dependencies > enforced > module array. This adds an enforced dependency on your own module. The effect is that the migration will be removed from Drupal’s active configuration when your custom module is uninstalled. Note that the top level dependencies array can have others keys in addition to enforced. For example: config and module. Learning more about them is left as an exercise for the curious reader.
It is important not to confuse the dependencies and migration_dependencies options. The former is provided by Drupal’s configuration management system and was just explained. The latter is provided by the Migrate API and is used to declare migrations that need be imported in advance. Read this article to know more about this feature. The following snippet shows an example:
uuid: b744190e-3a48-45c7-97a4-093099ba0547
id: udm_config_json_source_node_local
label: 'UD migrations configuration example'
dependencies:
enforced:
module:
- ud_migrations_config_json_source
migration_dependencies:
required:
- udm_config_json_source_image
- udm_config_json_source_paragraph
optional: []What did you learn in today’s blog post? Did you know that you can manage migrations in two ways: code or configuration? Did you know that file name and location as well as workflows need to be adjusted depending on which approach you follow? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.
Next: Workflows and benefits of managing Drupal migrations as configuration entities
This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.
Portside
Amplificando diversas voces a la izquierda.
Hi friends and collaborators, join us today at 3pm ET (or any subsequent Thursday at 3) as we kick off a series of research, planning, discussion, and building sessions for Visions Unite.
As our primary pro bono project, Agaric is working on Visions Unite, "where people seeking to make the world more whole can share ideas and information and gather the commitment and resources to build power to be the change we need", which a dozen projects have tried to do—what makes this different is sharing power via democratic mass communication.
Here are some initial user stories for Visions Unite.
Help plan and build the interface and underlying technology! (Drupal friends, we have been leaning against Drupal but might do it for the MVP— would love to hear your thoughts for or against.)
Connection info will always be up-to-date at agaric.coop/show (for these sessions we are taking over most of our Show & Tell hour, which is weekly on Thursdays 3pm Eastern).
TL;DR: For PHP Hexadecimals, Decimals and Octals are all Integers, so they must be declared as @param integer
While I was working on a patch I had to write the docblock of a function which received a hexadecimal number and I wasn't sure what I was supposed to put in the @type param.
I went to Drupal's API documentation and comments standards page to see which is the best type for this param and I found the following:
Data types can be primitive types (int, string, etc.), complex PHP built-in types (array, object, resource), or PHP classes.
Alright, a hexadecimal number is not a complex PHP built-in type nor a PHP Class so it must be a primitive type, so I went to the PHP documentation page to see which primitives PHP has and I found the following:
- boolean
- integer
- float (floating-point number, aka double)
- String
So there wasn't a specific reference for a Hexadecimal number...
The solution:
In the end Pieter Frenssen helped me (Thanks!) with this, and he showed me that in PHP, it doesn't matter what the base number is and it can be an octal, hexadecimal or a decimal, for PHP they all are integers (which makes sense but I wanted to be sure) and he shared this small snippet where we can see that PHP sees the numbers as integers and the base doesn't matter:
$ php -a Interactive shell php > var_dump(gettype(0x0f)); string(7) "integer" php > var_dump(0x08 === 8); bool(true)
So if you are writing the documentation of a function in which one of its params is a hexadecimal number you must declare it as Integer.
