Simplified Scheduler Objects

From Daxtoolkit
Jump to: navigation, search

In our initial design considerations for Dax, we envisioned that the control environment would have a single scheduler class or function that would handle the complication of launching a worklet in the execution environment. The same scheduler instance could be used over and over for worklet launching.

Although our current implementation mirrors this idea, there have been some complications. The first complication is that every type of worklet we have designed so far has needed its own custom invocation. Even a WorkletMapCell is slightly different than a WorkletMapField. The second complication is that many of the invocation methods really need to hold some state. For example, the scheduler for WorkletGenerateTopology needs to hold a “classification” array out of bounds of the invocation, and it generates a point mask that is optionally used to remove unused points after the fact. The third complication is that because we are using templating to get to the correct class for the scheduler, we end up creating and deleting custom schedulers under the cover for each invocation. So although it looks to the user like there is one instance of schedule used everywhere, under the covers the scheduler is still created and destroyed every time. This has not shown to be a high overhead, but still contrary to the initial idea.

The current implementation still does all this through a single Scheduler class, but also requires a rather complicated support structure of other classes. This proposal is to hopefully simply the implementation for both the internal Dax programmer and the Dax user.

When I wrote this, I didn't realize I had already made a design document for a scheduler redesign several months ago. The original design is at Generic Scheduler. This new design is not much different, but since writing the original document we have the ParameterPack class and know what it can do. --Kenneth Moreland 11:25, 22 October 2013 (EDT)

Current Implementation

As stated earlier, all scheduling is interfaced through a single Scheduler class. However, this is a rather small class that uses a large set of classes under the covers. First, there are “helper” classes to hold state required before and after the invocation. Second, there is an entire package, dax::cont::scheduling, that specializes the invocation to the particular type of worklet.

Helper Classes

We initially thought that the invocation of worklets would be stateless; all the data required would be passed in as parameters for the invocation (for example, fields) or as state of the worklet class. However, several of the invoke methods require their own input and output data that is best characterized as state for the invocation.

Currently, this state is held using “helper” classes that wrap around the worklet. Helper is not a great name, but we don’t really have a good name for them. All the helper classes take as a template parameter the type of worklet (and typically other parameters as well). This object has ivars for an instance of the worklet it is wrapping and any other pieces of state (such as count arrays, masks, indices, etc.).

This all provides a major inconvenience to the Dax user, who has to instantiate yet another object in order to schedule worklets. It is also confusing, and the fact that there is no naming scheme for these helper objects (because we could not think of one) does not help. To make matters worse, if you forget to use the helper class, the compiler is likely to give you a baffling error.

Specialized Schedulers

Although there is a single Scheduler class in the dax::cont namespace, this class is really just a thin facade over a large number of schedulers in dax::cont::scheduling. To date, every type of worklet (represented by a class named dax::exec::Worklet*) has its own scheduler. This works by using a trait class called dax::cont::scheduling::DetermineScheduler that produces a tag that is then used as a template parameter for dax::cont::scheduler::Scheduler. All scheduler implementations have a specialization for this class to implement the scheduling.

Although this mechanism should be completely hidden from the typical Dax user (other that trying to interpret errors), it is a pain for the scheduler implementer. In addition to writing a class designed to implement a particular scheduling, you have to add tags and compile-time comparisons throughout the DetermineScheduler header, all of which is tricky to debug.

Simplified Implementation

The scheduler redesign is intended to remove most of the complexity of the system by combining features. The basic idea is that all features for scheduling are combined into a single class.

First, there will be a base superclass for all Scheduler classes. It is called SchedulerBase and is hidden in the dax::cont::internal package. It also uses the curiously recurring template pattern so that it can forward invocations to the actual scheduler classes. The basic implementation looks as follows.

template<typename DerivedScheduler,
         typename WorkletBaseType,
         typename DeviceAdapter>
class SchedulerBase
{
public:
  template<typename WorkletType, typename ...T>
  DAX_CONT_EXPORT
  void Invoke(WorkletType w, T... arguments)
  {
    typedef boost::is_base_of<WorkletBaseType, WorkletType>
        Worklet_Should_Match_Scheduler;

    // If you get a compile error on the following line, then you are
    // using the wrong type of Scheduler class with your worklet.
    // (Check the type for WorkletType. It should match WorkletBaseType.)
    BOOST_MPL_ASSERT((Worklet_Should_Match_Scheduler));

    static_cast<DerivedScheduler*>(this)->DoInvoke(
      dax::internal::make_ParameterPack(arguments...));
  }

protected:
  template<typename WorkletType, typename ParameterPackType>
  DAX_CONT_EXPORT
  BasicInvoke(WorkletType worklet, ParameterPackType arguments) const
  {
  // Same implementation for the Invoke of the current implementation
  // of Invoke in SchedulerDefault.h
  ...
  }
};
May want to have both a const and non-const version of Invoke function. For stateless calls, you might want to do SchedulerFoo<>()::Invoke(worklet,a1,a2,...). However, if the invoke changes the state of the scheduler, the compiler won’t let you do this for a non-const Invoke function (I think). So, if the derived worklet does not change state, it could implement a const DoInvoke and both Invoke methods will work. If it does change state, it could implement a non-const DoInvoke and thus the non-const Invoke will work, but using the const version will cause a compile error. --Kenneth Moreland 11:25, 22 October 2013 (EDT)
The BasicInvoke command may need an optional count parameter to define how many instances to create. Most of the time, the BasicInvoke is able to get the number of instances by finding a parameter of the right domain. However, there is a case in ScheduleMapCell where the size has to come from the topology rather than the field. Then again, maybe it would be just as well for ScheduleMapCell to insert a dummy argument that matches the domain. --Kenneth Moreland 11:25, 22 October 2013 (EDT)
I am not confident that passing the worklet base class as a template parameter and asserting on Worklet_Should_Match_Scheduler is going to work. I suspect we will want to base some schedulers on other schedulers. For example, the scheduler that can generate geometry and find coincident points may want to use the key-value schedulers. However, with this design the key-value scheduler would assert that the scheduler is wrong. Maybe a way around this would be to have one scheduler able to call the BasicInvoke of another scheduler. --Kenneth Moreland 11:25, 22 October 2013 (EDT)

SchedulerBase contains two important methods. First, the Invoke method is a public method that behaves just like the Invoke method of the current dax::cont::Scheduler. Its only operations are to do a compile-time check to make sure the Scheduler and Worklet match, combine the arguments into a ParameterPack, and then pass that to a DoInvoke method of the derived class.

The second method is a protected method named BasicInvoke that derived classes can use to perform the actual scheduling. In the current scheduler implementation, almost all the scheduler types hold a reference to a default scheduler. In the new design, the scheduler class will simply call the BasicInvoke in their superclass.

The user-accessible scheduler classes will be in the dax::cont namespace and named Scheduler<type> where <type> matches the same name as in the worklet. For example, a worklet inheriting from WorkletGenerateKeysValues will be scheduled with a scheduler called SchedulerGenerateKeysValues.

Implementing a scheduler is straightforward. It has a public superclass of SchedulerBase and implements DoInvoke, which will do any “special” schedule operations and use the BasicInvoke to complete it. The implementation for SchedulerMapField, which does not do anything beyond the basic invoke, looks like the following.

template<typename DeviceAdapter = DAX_DEFAULT_DEVICE_ADAPTER_TAG>
class SchedulerMapField
  : public internal::SchedulerBase<SchedulerMapField<DeviceAdapter>,
                                   dax::exec::WorkletMapField,
                                   DeviceAdapter>
{
private:
  typedef internal::SchedulerBase<SchedulerMapField<DeviceAdapter>,
                                   dax::exec::WorkletMapField,
                                   DeviceAdapter> Superclass;
  friend Superclass;

  template<typename WorkletType, typename ParameterPackType>
  void DoInvoke(WorkletType worklet, ParameterPackType arguments) const
  {
    this->BasicInvoke(worklet, arguments);
  }
};

Other schedulers are implemented in roughly the same way, except that their DoInvoke method will be expanded to implement any additional scheduling requirements. For example, DoInvoke for the SchedulerGenerateTopology will perform the requisite scan, upper bounds, and visit indices and modify the parameters to send to the BasicInvoke.

Perhaps it makes more sense for BasicInvoke to be templated on Invocation so that the signatures can be set in the Invocation rather than having to create derived worklets that do nothing other than modify signatures. --Kenneth Moreland 11:25, 22 October 2013 (EDT)

Supplemental Design

Kenneth Moreland 17:55, 12 December 2013 (EST)

After today's weekly meeting, there was an impromptu design brainstorm that created some good suggestions for further redesign.

First, Robert suggested that we change the name to something like dispatch or dispatcher. This is a bit more indicative of what the class does. Schedule implies that it will make things happen at some indeterminate point in the future.

Second, I suggested that it might be a good idea to template the dispatcher classes based on the worklet type and hold an instance of the worklet. Then this instance will be used automatically when Invoked. A big advantage of this approach is that the calling structure of the Invoke method will exactly match the ControlSignature of the worklet rather than having one parameter added. I think this will be easier to understand and less prone to error. Also, this will allow the dispatcher to contain all the state needed to Invoke itself, so that it may be passed around to generic functions and Invoked without having to know what type of dispatcher to use.

So with this change, the DispatcherBase would look something like this:

template<typename DerivedDispatcher,
         typename WorkletBaseType,
         typename WorkletType,
         typename DeviceAdapter>
class DispatcherBase
{
public:
  template<typename ...T>
  DAX_CONT_EXPORT
  void Invoke(T... arguments)
  {
    static_cast<DerivedScheduler*>(this)->DoInvoke(
      this->Worklet, dax::internal::make_ParameterPack(arguments...));
  }

protected:
  DAX_CONT_EXPORT
  DispatcherBase(WorkletType worklet) : Worklet(worklet)
  {
    typedef boost::is_base_of<WorkletBaseType, WorkletType>
        Worklet_Should_Match_Scheduler;

  // If you get a compile error on the following line, then you are
  // using the wrong type of Scheduler class with your worklet.
  // (Check the type for WorkletType. It should match WorkletBaseType.)
  BOOST_MPL_ASSERT((Worklet_Should_Match_Scheduler));
  }

  template<typename WorkletType, typename ParameterPackType>
  DAX_CONT_EXPORT
  BasicInvoke(WorkletType worklet, ParameterPackType arguments) const
  {
  // Same implementation for the Invoke of the current implementation
  // of Invoke in SchedulerDefault.h
  ...
  }

private:
  WorkletType Worklet;
};

And the implementation of DispatcherMapField would look something like this:

template<typename WorkletType,
         typename DeviceAdapter = DAX_DEFAULT_DEVICE_ADAPTER_TAG>
class DispatcherMapField
  : public internal::DispatcherBase<DispatcherMapField<WorkletType,DeviceAdapter>,
                                    dax::exec::WorkletMapField,
                                    WorkletType,
                                    DeviceAdapter>
{
private:
  typedef internal::DispatcherBase<DispatcherMapField<WorkletType,DeviceAdapter>,
                                    dax::exec::WorkletMapField,
                                    WorkletType,
                                    DeviceAdapter> Superclass;
  friend Superclass;

  template<typename WorkletType, typename ParameterPackType>
  void DoInvoke(WorkletType worklet, ParameterPackType arguments) const
  {
    this->BasicInvoke(worklet, arguments);
  }

public:
  DispatcherFieldMap() : Superclass(WorkletType()) {  }
  DispatcherFieldMap(WorkletType worklet) : Superclass(worklet) {  }
};

Third, it was brought up that the helper classes generally have another template parameter that is the type for things like count arrays. That should be no problem. Just add a template parameter. Here is partially what DispatcherGenerateTopology would look like.

template<typename WorkletType,
         typename CountHandleType = dax::cont::ArrayHandle<dax::Id>,
         typename DeviceAdapter = DAX_DEFAULT_DEVICE_ADAPTER_TAG>
class DispatcherGenerateTopology
  : public internal::DispatcherBase<DispatcherMapField<WorkletType,CountHandleType,DeviceAdapter>,
                                    dax::exec::WorkletMapField,
                                    WorkletType,
                                    DeviceAdapter>
{
private:
  typedef internal::DispatcherBase<DispatcherGenerateTopology<WorkletType,CountHandleType,DeviceAdapter>,
                                    dax::exec::WorkletMapField,
                                    WorkletType,
                                    DeviceAdapter> Superclass;
  friend Superclass;

public:
  DispatcherFieldMap(CountResultType count)
    : Superclass(WorkletType()),
      RemoveDuplicatePoints(true),
      ReleaseCount(true),
      Count(count),
      PointMask()
  {  }

  DispatcherFieldMap(CountResultType count, WorkletType worklet)
    : Superclass(worklet)
      RemoveDuplicatePoints(true),
      ReleaseCount(true),
      Count(count),
      PointMask()
  {  }

  // All those set/get methods in GenerateTopology.

private:
  bool RemoveDuplicatePoints;
  bool ReleaseCount;
  CountResultType Classification;
  PointMaskType PointMask;

  template<typename WorkletType, typename ParameterPackType>
  void DoInvoke(WorkletType worklet, ParameterPackType arguments) const
  {
    // Implementation eventually involving this->BasicInvoke(worklet, arguments);
  }
};

Acknowledgements

Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-AC04-94AL85000.

SandiaLogo.png DOELogo.png

SAND 2013-9115P