Benjamin lives and works to connect people, ideas, and resources so more awesome things happen.

A web developer well-established with Drupal and PHP, he has also been enjoying programming projects with Django and Python. His work with Agaric clients has included universities (MIT and Harvard University), corporations (Backupify and GenArts), and not-for-profit organizations (Partners In Health and National Institute for Children's Health Quality).

Benjamin tries to aid struggles for justice and liberty, or try not to do harm in the meantime. He is an organizer of the  People Who Give a Damn, recognized as an incorporated entity by the IRS and the Commonwealth of Massachusetts charities division. He has also supported several artistic and philanthropic ventures and was a founding, elected director to the Amazing Things Arts Center in Framingham, Massachusetts.

He led 34 authors in writing the 1,100 page Definitive Guide to Drupal 7, but he is probably still best known in the Drupal community for posting things he finds to data.agaric.com where developers running into the same challenges find, if not answers, comfort that they are not alone.

You can get more (too much) of ben on a cooperatively-run part of the fediverse at @mlncn@social.coop.

Today we are going to learn how to migrate users into Drupal. The example code will be explained in two blog posts. In this one, we cover the migration of email, timezone, username, password, and status. In the next one, we will cover creation date, roles, and profile pictures. Several techniques will be implemented to ensure that the migrated data is valid. For example, making sure that usernames are not duplicated.

Although the example is standalone, we will build on many of the concepts that had already been covered in the series. For instance, a file migration is included to import images used as profile pictures. This topic has been explained in detail in a previous post, and the example code is pretty similar. Therefore, no explanation is provided about the file migration to keep the focus on the user migration. Feel free to read other posts in the series if you need a refresher.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD users whose machine name is ud_migrations_users. The two migrations to execute are udm_user_pictures and udm_users. Notice that both migrations belong to the same module. Refer to this article to learn where the module should be placed.

The example assumes Drupal was installed using the standard installation profile. Particularly, we depend on a Picture (user_picture) image field attached to the user entity. The word in parenthesis represents the machine name of the image field.

The explanation below is only for the user migration. It depends on a file migration to get the profile pictures. One motivation to have two migrations is for the images to be deleted if the file migration is rolled back. Note that other techniques exist for migrating images without having to create a separate migration. We have covered two of them in the articles about subfields and constants and pseudofields.

Understanding the source

It is very important to understand the format of your source data. This will guide the transformation process required to produce the expected destination format. For this example, it is assumed that the legacy system from which users are being imported did not have unique usernames. Emails were used to uniquely identify users, but that is not desired in the new Drupal site. Instead, a username will be created from a public_name source column. Special measures will be taken to prevent duplication as Drupal usernames must be unique. Two more things to consider. First, source passwords are provided in plain text (never do this!). Second, some elements might be missing in the source like roles and profile picture. The following snippet shows a sample record for the source section:

source:
  plugin: embedded_data
  data_rows:
    - legacy_id: 101
      public_name: 'Michele'
      user_email: 'micky@example.com'
      timezone: 'America/New_York'
      user_password: 'totally insecure password 1'
      user_status: 'active'
      member_since: 'January 1, 2011'
      user_roles: 'forum moderator, forum admin'
      user_photo: 'P01'
  ids:
    legacy_id:
      type: integer

Configuring the destination and dependencies

The destination section specifies that user is the target entity. When that is the case, you can set an optional md5_passwords configuration. If it is set to true, the system will take an MD5 hashed password and convert it to the encryption algorithm that Drupal uses. For more information password migrations refer to these articles for basic and advanced use cases. To migrate the profile pictures, a separate migration is created. The dependency of user on file is added explicitly. Refer to these articles more information on migrating images and files and setting dependencies. The following code snippet shows how the destination and dependencies are set:

destination:
  plugin: 'entity:user'
  md5_passwords: true
migration_dependencies:
  required:
    - udm_user_pictures
  optional: []

Processing the fields

The interesting part of a user migration is the field mapping. The specific transformation will depend on your source, but some arguably complex cases will be addressed in the example. Let’s start with the basics: verbatim copies from source to destination. The following snippet shows three mappings:

mail: user_email
init: user_email
timezone: user_timezone

The mail, init, and timezone entity properties are copied directly from the source. Both mail and init are email addresses. The difference is that mail stores the current email, while init stores the one used when the account was first created. The former might change if the user updates its profile, while the latter will never change. The timezone needs to be a string taken from a specific set of values. Refer to this page for a list of supported timezones.

name:
  - plugin: machine_name
    source: public_name
  - plugin: make_unique_entity_field
    entity_type: user
    field: name
    postfix: _

The name, entity property stores the username. This has to be unique in the system. If the source data contained a unique value for each record, it could be used to set the username. None of the unique source columns (eg., legacy_id) is suitable to be used as username. Therefore, extra processing is needed. The machine_name plugin converts the public_name source column into transliterated string with some restrictions: any character that is not a number or letter will be converted to an underscore. The transformed value is sent to the make_unique_entity_field. This plugin makes sure its input value is not repeated in the whole system for a particular entity field. In this example, the username will be unique. The plugin is configured indicating which entity type and field (property) you want to check. If an equal value already exists, a new one is created appending what you define as postfix plus a number. In this example, there are two records with public_name set to Benjamin. Eventually, the usernames produced by running the process plugins chain will be: benjamin and benjamin_1.

process:
  pass:
    plugin: callback
    callable: md5
    source: user_password
destination:
  plugin: 'entity:user'
  md5_passwords: true

The pass, entity property stores the user’s password. In this example, the source provides the passwords in plain text. Needless to say, that is a terrible idea. But let’s work with it for now. Drupal uses portable PHP password hashes implemented by PhpassHashedPassword. Understanding the details of how Drupal converts one algorithm to another will be left as an exercise for the curious reader. In this example, we are going to take advantage of a feature provided by the migrate API to automatically convert MD5 hashes to the algorithm used by Drupal. The callback plugin is configured to use the md5 PHP function to convert the plain text password into a hashed version. The last part of the puzzle is set, in the process section, the md5_passwords configuration to true. This will take care of converting the already md5-hashed password to the value expected by Drupal.

Note: MD5-hash passwords are insecure. In the example, the password is encrypted with MD5 as an intermediate step only. Drupal uses other algorithms to store passwords securely.

status:
  plugin: static_map
  source: user_status
  map:
    inactive: 0
    active: 1

The status, entity property stores whether a user is active or blocked from the system. The source user_status values are strings, but Drupal stores this data as a boolean. A value of zero (0) indicates that the user is blocked while a value of one (1) indicates that it is active. The static_map plugin is used to manually map the values from source to destination. This plugin expects a map configuration containing an array of key-value mappings. The value from the source is on the left. The value expected by Drupal is on the right.

Technical note: Booleans are true or false values. Even though Drupal treats the status property as a boolean, it is internally stored as a tiny int in the database. That is why the numbers zero or one are used in the example. For this particular case, using a number or a boolean value on the right side of the mapping produces the same result.

In the next blog post, we will continue with the user migration. Particularly, we will explain how to migrate the user creation time, roles, and profile pictures.

What did you learn in today’s blog post? Have you migrated user passwords before, either in plain text or hashed? Did you know how to prevent duplicates for values that need to be unique in the system? Were you aware of the plugin that allows you to manually map values from source to destination? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Migrating users into Drupal - Part 2

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Agaric builds tools for medical and scientific communities to advance their work, enhance collaboration, and improve outcomes.  And we've been doing this—helping healthy discussion about science and medicine flourish online—since 2008.

Reusable platforms for advanced online collaboration

Therapy Fidelity App

Agaric is developing the Therapy Fidelity app: an all-in-one, inexpensive mobile web application to help therapists do the work of counseling. The app automates surveys, handles multiple CBT protocols, tracks fidelity, monitors outcomes, and more. This application is being developed for Scheeringa Mind Company with initial funding by Tulane University.  It is built in JavaScript (React and Typescript) and Golang and makes extensive use of Truevault and AWS APIs.

The Collaboratory

The National Institute for Children's Health Quality partnered with Agaric to build the Collaboratory—a platform designed specifically to help healthcare improvement teams collaborate, innovate, and make change. During this partnership, begun in 2015, Agaric built a collaborative analytics tool that allows healthcare quality teams to visualize, compare, and benchmark data, identify opportunities for improvement, and celebrate their successes. We were proud to be NICHQ's 2020 partners in making the most of the digital health revolution.

Platform for Experimental Collaborative Ethnography (PECE)

In 2015, we began contributing to PECE, an open source digital platform that supports multi-sited, cross-scale ethnographic and historical research.  PECE is built as a Drupal distribution that can be improved and extended like any other Drupal project.  Agaric's contributions include building an API integration between PECE's bibliographic citation capabilities and Zotero's open source reference management and collaborative bibliography tools.

We have been brought back for a larger role to realize the full upgrade of this distribution and platform to Drupal 10.

Partners In Health Drug Resistant Tuberculosis Network and EndTB.org

Beginning in 2010, Agaric took over the development of the DRTB Network platform for Partners In Health, the famed international nonprofit public health organization, and the TB Care II initiative.  The core of this work was connecting practitioners in the field with experts through a natural yet structured response process complete with careful editorial review.  This crucial work lives on with endTB.org, a partnership between Partners In Health, Médecins Sans Frontières, and Interactive Research & Development.  All of this work for PIH is in Drupal.

Science Collaboration Framework

Agaric was the lead developer for the Science Collaboration Framework, a project of Harvard's Initiative in Innovative Computing. Working with researchers from Harvard University and Massachusetts General Hospital, we built a reusable platform for collaboration and communication in biomedical research, enriching the contributions of scientists and the biomedical online community with semantic data, highlighting advanced, structured relationships between contributed resources, and facilitating structured community discourse around biomedical research. We even earned a writeup in a scientific journal for our work!

As part of SCF, Agaric led the work of building the website for an online community of Parkinson's disease researchers and research investors, on the Science Collaboration Framework, for the Michael J. Fox Foundation for Parkinson's Research.

Ask Agaric for help building your world-bettering community today!

We have a number of processes and tools we use as we develop:

• Local development: we all develop locally on our own computers and push each change to a common code repository which we maintain.

• Each task including new features, bugs or changes are kept track of as an issue ticket. These tickets will be accessible to Urban Edge staff as well as ourselves. Here is where we can track progress in code changes, elaborate on details and generally where we keep track of the hundreds of small issues that will come up.

• Resources allowing, we use automated tests which emulate a human's interaction with a browser to make sure new work we does not unknowingly break site functionality. This ensures a solidly functioning site even as major changes take place.

• We have experience managing a wide range of servers, server environments, or specialized hosting environments, and related technologies:

  • We have written our own battle-tested server deployment code, which securely pushes changes in code to any target location.
  • Deployments can run manually or to a desired level of automation, such as always deploying to a staging website automatically on a successful build, and then manually progressing to live after review.
  • Our existing deployment tools can handle configuring a virtual (or physical) server from the ground up, or else function within a variety of PHP-based website hosting providers.
  • Additionally, we can support container-based environments by providing docker container configurations for running the required services.
  • Dependent upon the hosting platform, we also manage additional security and reliability features, such as sites having automated system updates, virus scans of uploaded files, custom backup schedules, additional firewall protections, monitoring services, or other customized needs.
  • We can also work alongside existing IT service providers with recommendations regarding things you may already manage, such as CDN (content delivery networks), email, or DNS (domain name) changes that may be needed.

Read more about our tools for environment management and development.

Welcome to Drutopia! We hope you are enjoying the features we have built. Everything you are using is open-source and free for good people like yourself to use.

We invite you to give your input on the project and contribute where you can by becoming a member of Drutopia. For as little as $10 a year you can become a member who votes for our leadership team, suggests features for our roadmap and is part of a community building tools for the grassroots.

Learn more about membership at drutopia.org

Blog posts can be a valuable way for a variety of authors to post reflections, experiences and opinions.

At a minimum every blog post should have an image (which will be associated with the blog and display in different sizes in different displays), a summary field and a text paragraph in the body field.

All Drutopia content types are built using the Paragraphs module which lets you add a variety of types of content and arrange them in the order you wish. When creating a blog post you can add standard text paragraphs in the body paragraph field, but also add images or files (such as a PDF).

You can define your own permissions for the Drupal permissions page (/admin/people/permissions in modern Drupal, Drupal 8, 9, 10, and beyond) and then add conditional options to your code to do different things based on the role of the user and the permissions configured by a site administrator.

Here's how.

Create a modulename.permissions.yml file

This simple file has the permission machine name (lower case with spaces) and a title (Sentence case) with an optional description.

For our module, which has a particularly long name, that file is drutopia_findit_site_management.permissions.yml and its contents are like so:

access meta tab:
  title: 'Access meta tab'
  description: 'Access meta information (author, creation date, boost information) in Meta vertical tab.'

You can repeat lines like these in the same file for as many permissions as you wish to define.

Check for that permission in your code

The process for checking permissions is simply to use a user object if that's handed into your code, or to load the current user if it's not, and use the hasPermission() method which returns TRUE if that user has permission and FALSE if not.

For example, in a form alter in our drutopia_findit_site_management.module file:

/**
 * Implements hook_form_BASE_FORM_ID_alter() for node_form.
 *
 * Completely hide the Meta vertical tab (field group) from people without permission.
 *
 */
function drutopia_findit_site_management_form_node_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  // If the current user has the permission, do not hide the Meta vertical tab.
  if (\Drupal::currentUser()->hasPermission('access meta tab')) {
    return;
  }
  // Code to hide the meta tab goes here, and is only reached if the user lacks the permission. 
  // ...
}

See all this code in context in the Find It Site Management module.

To learn more about defining permissions in modern Drupal, including dynamic permissions, you can see the change record for when the new approach replaced hook_permission().

Thinking it would be a great place to work a day or two while in New York City for clients or DrupalCamps, Agaric dropped a few dollars in the Kickstarter fund for New Work City: Community Coworking Center for Independents in NY.

On November 13^th and 14^th in New York City, several hundred people gathered to talk about the problems of an online economy reliant on monopoly, extraction, and surveillance—and discuss how to build a "cooperative Internet, built of platforms owned and governed by the people who rely on them."

My experience at the Platform Cooperativism summit was Wow, everyone here really gets it and so many are doing awesome things; and then Hmm, there are still some really important differences to be worked out; and then We'll have to continue for months to figure out strategy for building fair platforms and we also need to restructure the whole economy.

Some definitions

In the sense technologists use it a platform is, like a physical platform, a technology that holds a lot of people up. It convenes people and gives them a chance to do something they wouldn't otherwise be able to do. Platforms can often be natural monopolies due to capturing the benefits of network effects (one person with a telephone is pointless, having nearly everyone available by telephone is incredibly valuable). Amazon and eBay are both platforms for sellers and buyers, Uber and Lyft for drivers and riders, Mechanical Turk and TaskRabbit for piece-workers and buyers of their work.

A cooperative is a jointly owned and democratically-controlled enterprise formed by people voluntarily uniting to meet their common needs and aspirations. Agaric is a small worker-owned cooperative, Mondragon is a very large group of integrated worker cooperatives, consumer cooperatives are businesses owned by their customers, credit unions are financial institutions owned by their members (with a one person, one vote governance), and producer cooperatives like CROPP Cooperative are formed by member businesses (which are not necessarily cooperatives themselves).

A platform cooperative, then, is a platform owned and controlled by the people directly affected by it. A company must be accountable, and as Omar Freilla put it, accountable means those impacted make the decisions.

The power to do harm

This summit was a follow-up to the Digital Labor summit held one year before which detailed myriad ways centralized online platforms extract value from dispersed workers who have few options or bargaining power. Control of online platforms by the representatives of capital has or will have negative effects on workers, similar to exploitation in global manufacturing (think electronic devices and clothing), and negative effects on customers (think the massive money grab by oligopolies of fossil fuel and telecommunications corporations).

Agaric's Michele Metts told the Digital Labor summit organizers every chance she got that cooperatives and Free Software were the answer to exploited labor in the Internet economy, but something even more powerful than Micky's advocacy must have been at work: nearly every participant at Platform Cooperativism spoke of the need for workers to own the platforms that control their work, and people presenting on technology took for granted that source code and algorithms have to be open for democratic control to be meaningful. As Micky said on her panel, "You cannot build a platform for freedom on someone else's slavery."

The opening presentations made the case that platforms will exploit us unless we take control, and we moved on to discussing strategies for building platform businesses that are cooperatives of the people using the platforms. We also celebrated those already starting, like Loconomics, Fairmondo (in Germany), and Member's Media.

Building for shared power and uplift

The biggest unsolved, but acknowledged, problem is getting the resources to build platforms that can compete with venture capital-funded platforms. Dmytri Kleiner made the claim that profit requires centralization, and, moreover, that centralization requires organizing along the lines of a profit-taking venture. How can people get the resources to build without both having to give up control and having to exploit people using the platform? Robin Chase reminded us that it costs millions of dollars, at least, to build a viable platform. Her solution is to continue to seek venture capital and work for some environmental or community goals while compromising on control.

A more popular possible solution is to replace centralized systems with decentralized ones, even to the point of replacing specific software with protocols, so the cost of building and operating platforms can be more widely shared, along with the benefits. However, as Astra Taylor summed up the widely felt point, decentralization does not always mean distributed power. Therefore control of technology decisions, and so democratic control of platforms, is more important than technology itself.

The potential positive role for government regulation was often mentioned, as Sarah Ann Lewis summarized the sentiment in a tweet: Platforms are not special snowflakes that must be exempt from regulation. If you can only succeed by exploitation you deserve to melt. Indeed, the centralized and surveillance nature of most platforms would make it much easier to ensure non-discrimination and fair wages.

More excitement came from the mention that local government has long played a role and can play a stronger part in democratic ownership of physical spaces. Several speakers urged people to get involved in local government, where harmful policies may be more the result of a lack of knowledge than of embedded corruption. Government can also get involved in mandating an open API for ride hailing services, which would remove the monopoly power from centralizing companies.

On ownership and control

Hundreds of possible solutions faced lively questioning and debate, yet in all of this the titular solution, cooperative ownership, did not get the scrutiny it merits. Jessica Gordon Nembhard's Collective Courage has made me see that the connections and overlaps between worker cooperatives and other types of cooperatives are much more significant than I'd thought, but there are still differences. These differences, and the need to decide who exactly is democratically controlling a platform, were often not made clear by presenters, including some who are building platform cooperatives.

If Stocksy, for example, is owned by its photographers, can the workers who build the platform technology (rather than use it) play a part in democratic control? Co-founder and CEO Brianna Wettlaufer refers to it as a multi-stakeholder cooperative and it has been around since 2012 so they've surely worked it out, but this question is at the heart of how platform cooperatives must operate and it was hardly addressed at all.

The answer can be simple. The Black Star Coop brewery and restaurant in Austin, Texas, is owned by its customer-members while the workers manage it. The workers are internally a democracy, but there's no question they work for a businesses which is managed democratically by the customers. This makes even more sense for a quasi-monopoly platform: It's more important for, say, millions of people relying on a platform for livelihood or transportation or communication to own it than for the relatively small number of people who built it to own it.

This brings up another question that went largely unasked at the conference: does ownership mean anything when it's spread out among thousands or millions of people? Federated structures can mitigate this, but in general whoever controls communication among members effectively controls decisions. It may be possible to have horizontal mass communication by way of democratic moderation. At a small workshop I held at the conference, participants discussed ways collective control can be made real as democratic platforms scale—but that's a topic for another discussion.

The sense that displacing an app or website is easier than reconstructing global supply chains fueled a lot of the excitement at the conference. Notwithstanding, the need to restructure the rest of the economy so that it works to serve the needs of people, rather than sacrificing people's needs to the dictates of the economy, was never far from people's minds. Videos of most sessions are online and will certainly make you think about the opportunities for cooperative ownership of services and structures that define our lives, online and off.

Thank you for requesting that Micky Metts speak at your event. She will respond as soon as she can and let you know if she is available, or if she has any questions.

Agaric Team

Learning online has become a necessity for schools and independent teachers and tutors. Agaric can provide a Learning Management System (LMS) to support your teaching skills and knowledge integrated with video chat and whiteboard features. Every component of our platform is 100% Free/Libre Software because we believe in protecting the privacy of our students!

Let us know what you need and we can set up a time to discuss your unique situation.

Here you will find the details of our online learning tools.

Learning Management System

The Canvas learning management platform allows you or your organization to build the digital learning environment that meets the unique challenges faced by your institution. Canvas LMS allows teachers to create courses and grade students. Developed in 2011, Canvas was designed to better engage people in teaching and learning processes. 

Canvas is integrated with video chat software called BigBlueButton and it works to seamlessly allow teachers and students, if given the permission, to create live conferences with many features for moderation and engagement for all participants. Both Canvas and BigBlueButton are free open source software because our mission includes  privacy for all and the ability to freely modify the software to suit your needs.

Canvas LMS Features:

• Practice Tests, or Quizzes – These are assignments, groups of questions, that can be used to challenge student understanding and assess comprehension.  Question types include multiple choice, true/false, fill in the blanks, multiple answers, matching, numerical answers, formulas, and long text (freeform/essay).
• Question Banks – Group quiz questions in question banks to create grade, topic, institutional, or departmental question repositories.
• Outcomes – Outcomes (also called standards or competencies) describe what a learner should be able to do and are used to measure knowledge and ability.
• Mastery Paths – This tool allows course content to automatically be released to a learner based on performance, providing differentiation to students.
• Canvas Data – Canvas Data allows instructors to parse Canvas-generated data quickly.
• Canvas Commons – Canvas Commons is a repository that lets users easily share content.
• Integrated Media Recorder – This tool allows users to record audio and video messages for the digital classroom.
• Web conferencing – Set a meeting time for the whole class or one-on-one synchronous online discussions.
• Mastery Gradebook – Helps instructors assess the Outcomes being used in Canvas courses and measure student learning for accreditation or standards-based grading.
• Canvas Parent – Canvas Parent allows parents to engage with their children’s education by reviewing upcoming or past assignments, checking grades and receiving course announcements.
• Canvas Polls – An app (available on iOS and Android) that gauges students’ comprehension of material without “clicker” devices.

Videochat and whiteboard

The BigBlueButton video and whiteboard software is integrated into the Canvas dashboard and allows teachers and students to meet in a live learning environment that brings interactivity to the forefront. BigBlueButton is a free open-source web conferencing system designed for online classes, workshops and lectures. BigBlueButton allows teachers to create breakout rooms for different levels of learning.

Big Blue Button Features:

• Accounts and Profile
  • Sign up / Login
  • Profile
• Conference Rooms
  •  Create New Conference Rooms
  •  Breakout Rooms
  •  Conference Settings
  •  Rename Conference
  •  Manage Access to Conferences
•  Record Conference
  •  View Recordings
  •  Manage Recordings
  •  Modify Recordings
  •  Sort and Search Recordings

Drupal content management platform

You can provide a simple website for your school by building on the Drupal platform, the Canvas learning management platform automatically has access to a large number of features that are built into Drupal.  We describe some important ones here, though there are many more.

Content editing with Drupal

• Build your website with basic pages, landing pages, news postings, and more.
• Create instructional guides and reference links to relevant information.
• Engage the students in making additions to the site and maintaining the content.
• Optionally, use a content moderation workflow in which a content editor can approve draft content.
• Restore deleted content (retrieve from trash), so student exercises involving creating content on the site are not lost.
• Administrators can set permissions on any content and allow access in a granular way.

What will you provide for your students?