mm_modules_091_100.patch

diff --git a/Documentation/mm_modules.txt b/Documentation/mm_modules.txt
new file mode 100644
index 0000000..9afba9d
--- /dev/null
+++ b/Documentation/mm_modules.txt
@@ -0,0 +1,284 @@
+	Virtual (MM_MODULES) and Physical (PMEM_MODULES) modules for Linux 2.6.32
+
+	Gil Tene <gil@azulsystems.com>
+
+In order to support extended functionality for virtual and physical memory,
+enabling loadable modules to deliver integrated memory management
+functionality is desirable.
+
+Examples of valuable extended functionality can include:
+
+- Support for mappings with multiple and mixed page sizes
+	- Including transitioning of mapped addresses from large to small page
+	  mappings, or small to large.
+- Support for very high sustained mapping modification rates:
+	- Allowing concurrent modifications within the same address space
+	- Allowing user to [safely] indicate lazy TLB invalidation and
+	  thereby dramatically reduce per change costs
+	- Supporting fast, safe application of very large "batch" sets
+	  of mapping modifications (remaps and mprotects), such that
+	  all changes become visible within the same, extremely short
+	  period of time.
+- Support for large number of disjoint mappings with arbitrary manipulations
+  at high rates
+
+In order to support such functionality, memory management modules need
+to interact with several points in the virtual and physical memory systems
+that are "lower" that those of a typical module or driver under previously
+kernel interfaces. The specific interface points are itemized below.
+
+A set of proposed patches against the RHEL 2.6.32-rc7-git1 is provided,
+which creates the appropriate interfaces for module to register with,
+allowing them to interact with vmas, mm structures and pages through
+their needed lifecycle transitions, as well as keep state they
+may need associated with vmas and mm structures.
+
+----------------------------------------------------------------------
+
+At a high level, the patches represent:
+
+Changes to existing data structures:
+
+- An added "mm_modules" field to struct mm_struct.
+
+- Two added fields to "mm_module_ops" struct vm_area_struct.
+
+New data structures:
+
+- Four new common data types (struct mm_module_struct,
+  struct pmem_module_struct, struct pmem_module_operations_struct,
+  mm_module_operations_struct)
+
+Code changes:
+
+- changes to add calls into mm_module_ops and pmem_module_ops at
+  various appropriate locations.
+
+- changes to disable or make invalid certain operations (e.g. vma
+  split, merge, remap) for vmas that are controlled by mm_modules
+
+- A change to fault handling to allow handle_mm_fault to return
+  an indication for SEGV_MAPERR or SEGV_ACCERR (allow for sparsely
+  mapped, and non-homogeneously protected vmas).
+
+- A change to gup_fast (arch/x86/mm/gup.c) to make it safely independent
+  of any page table locking and invalidation schemes (as long as whatever
+  they do is safe in an SMP environment), including mechanisms that
+  may ref-count pages down to 0 before tlb-invalidating their mappings.
+
+----------------------------------------------------------------------
+Note: about need for physical memory support:
+
+While virtual memory functionality alone can support some of the 
+possible extended functionality, high performance functionality 
+requires physical memory management and control as well. A good example
+of this is in-process recycling of memory and in-process memory free
+lists and their use in dramatically dampening TLB invalidate requirements
+on allocation or deallocation edges. When a system need to sustain 
+a high rate of new mappings (e.g. 20GB/sec of sustained random, disjoint 
+map/remap/unmap operations), such in-process physical memory free lists
+become a must. 
+
+----------------------------------------------------------------------
+Note: About hugetlb
+
+To increase the likelihood of usefulness to generic virtual memory
+functionality additions, the module interface was designed such that
+the all current hugetlb functionality could be developed as a loadable
+kernel module under the proposed interface.
+
+----------------------------------------------------------------------
+
+Some high level design points:
+
+- Virtual memory modules (mm_modules) are generally responsible for
+creating and controlling their own vmas. [whole] vmas can be torn down 
+by the kernel.
+
+- The kernel's "normal" memory manipulation system calls will not modify
+the bounds of an mm_module managed vma. [i.e. no merging, no splitting,
+no remapping]. mm_modules may support such functionality through their own
+entry points.
+
+- mm_modules must adhere to the kernel's convention for locking the
+page table hierarchy for any part of the hierarchy that may be manipulated
+by other code. While mm_modules may apply private locking schemes
+to parts of the hierarchy (e.g. below the pmd level), they must do
+so only with parts of the hierarchy that are know to be completely owned
+by the module. [e.g. 2MB aligned vmas can separately control locking at
+the pmd level and below]
+
+- mm_modules can carry unique state per mm, and unique state per vma.
+
+- mm_modules provide their own fault handling functionality. They may
+  indicate a need to SEGV with a mapping or protection si_code (sparsely
+  mapped vas are a good example of this need).
+
+- pmem_modules manage their own lists of physical pages, and are expected
+  to be aware of physical pages that they are supposed to control, even 
+  when (and especially when) those pages carry a 0 ref count.. They can
+  do so in any way they want (e.g. a module-private vmemmap mirror, or
+  one using much larger aligned page sizes).
+
+- registered pmem_modules intercept all physical page releases at
+  put_page() and release_pages(), such that when a page is ref-counted down
+  to 0, the pmem_module would pick it up before it reaches the system's
+  normal free lists.
+
+- pmem_modules are expected to support hot_plug functionality. When physical
+  memory is added to the system, all current pmem_modules must adjust their
+  internal maps of physical memory to be able to correctly handle physical
+  pages of the newly discovered range.  
+
+----------------------------------------------------------------------
+Virtual Memory Module Interface Points:
+
+Fault handling: 
+	int (*handle_mm_fault)(struct mm_struct *mm,
+			struct vm_area_struct *vma, unsigned long addr,
+			int write_access);
+
+	Called from handle_mm_fault() for vmas managed by the mm_module to 
+	satisfy fault handling needs. May return an indication of SEGV_ACCERR
+	or SEGV_MAPERR if fault address is not mapped (e.g. for sparsely
+	populated vmas).
+
+Protection changes:
+	int (*change_protection)(struct vm_area_struct *vma, unsigned long start,
+			unsigned long end, unsigned long newflags);
+
+	Called from mprotect_fixup() to change the protection of all mapped
+	pages within a vma managed by the mm_module. [needed for e.g. hugetlb
+	interaction with mprotect()]. Note: the module may (and likely will)
+	provide it's own, finer grain protection control calls.
+
+Page range duplication:
+	int (*copy_page_range)(struct mm_struct *dst_mm,
+			struct mm_struct *src_mm, struct vm_area_struct *vma);
+
+	Called from copy_page_range() to duplicate a vma managed by an mm_module
+	from a src_mm to a dst_mm. Used for specialized forking behavior.
+
+Page following:
+	int (*follow_page)(struct mm_struct *mm, struct vm_area_struct *vma,
+			struct page **pages, struct vm_area_struct **vmas,
+			unsigned long *position, int *length,
+			int i, int write);
+
+	Called from get_user_pages() to get a vector of pages associated with a
+	range of addresses within a vma managed by the mm_module. Required
+	for core dumping, gdb, etc.
+
+Probe mapping protection and range:
+	int (*probe_mapped)(struct vm_area_struct *vma, unsigned long start,
+			unsigned long *end_range, unsigned long *range_vm_flags);
+
+	return an indication of whether an address within a vma is mapped or
+	not, along with it's protection and the range of identical protection
+	mapping. Used by core dump functionality (e.g. elf_core_dump()) for
+	efficient traversal and dumping of very large and sparsely populated
+	vmas (e.g. 16TB vma containing 300MB of mapped data).
+
+Unmapping:
+	unsigned long (*unmap_page_range)(struct mmu_gather **tlbp,
+			struct vm_area_struct *vma, unsigned long addr,
+			unsigned long end, long *zap_work,
+			struct zap_details *details); 
+
+	Called by unmap_vmas() to unmap and release all pages with a vma
+	managed by the mm_module.
+
+	void (*free_pgd_range)(struct mmu_gather *tlb, unsigned long addr,
+			unsigned long end, unsigned long floor,
+			unsigned long ceiling);
+
+	Called by free_pgtables() to tear down all page table hierarchy
+	storage associated with a vma managed by the mm_module.
+
+vma lifecycle:
+	int (*init_module_vma)(struct vm_area_struct *vma,
+			struct vm_area_struct *old_vma);
+
+	Called by dup_mmap() to initialize the mm_module state associated with
+	a newly duplicated vma managed by the mm_module.
+
+	void (*exit_module_vma)(struct vm_area_struct *vma);
+
+	Called by remove_vma() to tear down the mm_module state associated with
+	the vma managed by the mm_module.
+
+mm lifecycle:
+	int (*init_module_mm)(struct mm_struct *mm,
+			struct mm_module_struct *mm_mod);
+
+	Called from mm_init to initialize the mm_module state associated with a
+	newly duplicated mm.
+
+	int (*exit_module_mm)(struct mm_struct *mm,
+			struct mm_module_struct *mm_mod);
+
+	Called by mmput to tear down the mm_module state associated with an mm
+
+struct mm_module_operations_struct {
+	int (*handle_mm_fault)(struct mm_struct *mm,
+			struct vm_area_struct *vma, unsigned long addr,
+			int write_access);
+	int (*change_protection)(struct vm_area_struct *vma, unsigned long start,
+			unsigned long end, unsigned long newflags);
+	int (*copy_page_range)(struct mm_struct *dst_mm,
+			struct mm_struct *src_mm, struct vm_area_struct *vma);
+	int (*follow_page)(struct mm_struct *mm, struct vm_area_struct *vma,
+			struct page **pages, struct vm_area_struct **vmas,
+			unsigned long *position, int *length,
+			int i, int write);
+	int (*probe_mapped)(struct vm_area_struct *vma, unsigned long start,
+			unsigned long *end_range, unsigned long *range_vm_flags);
+	unsigned long (*unmap_page_range)(struct mmu_gather **tlbp,
+			struct vm_area_struct *vma, unsigned long addr,
+			unsigned long end, long *zap_work,
+			struct zap_details *details); 
+	void (*free_pgd_range)(struct mmu_gather *tlb, unsigned long addr,
+			unsigned long end, unsigned long floor,
+			unsigned long ceiling);
+	int (*init_module_vma)(struct vm_area_struct *vma,
+			struct vm_area_struct *old_vma);
+	void (*exit_module_vma)(struct vm_area_struct *vma);
+	int (*init_module_mm)(struct mm_struct *mm,
+			struct mm_module_struct *mm_mod);
+	int (*exit_module_mm)(struct mm_struct *mm,
+			struct mm_module_struct *mm_mod);
+};
+
+---------------------------------------------------------------
+Physical Memory Module Interface Points:
+
+Page release interception:
+	int (*put_page)(struct page *page);
+
+	Called by put_page() to allow a pmem_module to receive a released page
+	under it's management. Returns 1 if page was "taken" (determined to 
+	belong to the pmem_module), or 0 if not.
+
+	int (*release_page)(struct page *page, struct zone **zonep,
+			unsigned long flags);
+
+	Called by release_pages() to allow a pmem_module to receive a released
+	page under it's management. Returns 1 if page was "taken" (determined to 
+	belong to the pmem_module), or 0 if not. If page was taken, the spinlock
+	&(*zonep)->lru_lock must also be released as per similar behavior in
+	release_pages().
+
+Memory hotplug support:
+	int (*sparse_mem_map_populate)(unsigned long pnum, int nid);
+
+	Called by kmalloc_section_memmap() to allow the pmem_module to 
+	initialize page mapping state associated with newly discovered
+	physical memory. Must return 0 if not successful.
+
+struct pmem_module_operations_struct {
+	int (*put_page)(struct page *page);
+	int (*release_page)(struct page *page, struct zone **zonep,
+			unsigned long flags);
+	int (*sparse_mem_map_populate)(unsigned long pnum, int nid);
+};
+
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index f627779..94ee45b 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -1120,6 +1120,16 @@ good_area:
 	fault = handle_mm_fault(mm, vma, address, write ? FAULT_FLAG_WRITE : 0);
 
 	if (unlikely(fault & VM_FAULT_ERROR)) {
+#ifdef CONFIG_MM_MODULES
+		if (fault & (VM_FAULT_SIGSEGV)) {
+			if (fault & VM_FAULT_SEGV_ACCERR)
+				bad_area_access_error(regs, error_code,
+					address);
+			else
+				bad_area(regs, error_code, address);
+			return;
+		}
+#endif /* CONFIG_MM_MODULES */
 		mm_fault_error(regs, error_code, address, fault);
 		return;
 	}
diff --git a/arch/x86/mm/gup.c b/arch/x86/mm/gup.c
index 738e659..f25621b 100644
--- a/arch/x86/mm/gup.c
+++ b/arch/x86/mm/gup.c
@@ -62,6 +62,20 @@ retry:
 #endif
 }
 
+#ifdef CONFIG_MM_MODULES
+static inline int get_page_not_zero(struct page *page)
+{
+	page = compound_head(page);
+	return atomic_inc_not_zero(&page->_count);
+}
+
+static inline int get_head_page_multiple_not_zero(struct page *page, int nr)
+{
+	VM_BUG_ON(page != compound_head(page));
+	return atomic_add_unless(&page->_count, nr, 0);
+}
+#endif /* CONFIG_MM_MODULES */
+
 /*
  * The performance critical leaf functions are made noinline otherwise gcc
  * inlines everything into a single function which results in too much
@@ -88,7 +102,13 @@ static noinline int gup_pte_range(pmd_t pmd, unsigned long addr,
 		}
 		VM_BUG_ON(!pfn_valid(pte_pfn(pte)));
 		page = pte_page(pte);
+#ifdef CONFIG_MM_MODULES
+		/* indicate failure if page ref count was already 0 */
+		if (!get_page_not_zero(page))
+			return 0;
+#else /* !CONFIG_MM_MODULES */
 		get_page(page);
+#endif /* CONFIG_MM_MODULES */
 		pages[*nr] = page;
 		(*nr)++;
 
@@ -132,9 +152,18 @@ static noinline int gup_huge_pmd(pmd_t pmd, unsigned long addr,
 		page++;
 		refs++;
 	} while (addr += PAGE_SIZE, addr != end);
+#ifdef CONFIG_MM_MODULES
+	if (!get_head_page_multiple_not_zero(head, refs)) {
+		/* revert nr pages, indicate failure (ref count was 0) */
+		(*nr) -= refs;
+		return 0;
+	}
+	return 1;
+#else /* !CONFIG_MM_MODULES */
 	get_head_page_multiple(head, refs);
 
 	return 1;
+#endif /* CONFIG_MM_MODULES */
 }
 
 static int gup_pmd_range(pud_t pud, unsigned long addr, unsigned long end,
@@ -189,9 +218,18 @@ static noinline int gup_huge_pud(pud_t pud, unsigned long addr,
 		page++;
 		refs++;
 	} while (addr += PAGE_SIZE, addr != end);
+#ifdef CONFIG_MM_MODULES
+	if (!get_head_page_multiple_not_zero(head, refs)) {
+		/* revert nr pages, indicate failure (ref count was 0) */
+		(*nr) -= refs;
+		return 0;
+	}
+	return 1;
+#else /* !CONFIG_MM_MODULES */
 	get_head_page_multiple(head, refs);
 
 	return 1;
+#endif /* CONFIG_MM_MODULES */
 }
 
 static int gup_pud_range(pgd_t pgd, unsigned long addr, unsigned long end,
diff --git a/drivers/misc/sgi-gru/grufault.c b/drivers/misc/sgi-gru/grufault.c
index 38657cd..31e0efe 100644
--- a/drivers/misc/sgi-gru/grufault.c
+++ b/drivers/misc/sgi-gru/grufault.c
@@ -192,6 +192,10 @@ static int non_atomic_pte_lookup(struct vm_area_struct *vma,
 {
 	struct page *page;
 
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops)
+		return -EFAULT;
+#endif /* CONFIG_MM_MODULES */
 #ifdef CONFIG_HUGETLB_PAGE
 	*pageshift = is_vm_hugetlb_page(vma) ? HPAGE_SHIFT : PAGE_SHIFT;
 #else
@@ -251,6 +255,9 @@ static int atomic_pte_lookup(struct vm_area_struct *vma, unsigned long vaddr,
 #else
 	*pageshift = PAGE_SHIFT;
 #endif
+#ifdef CONFIG_MM_MODULES
+	*pageshift = (pmd_large(*pmdp)) ? HPAGE_SHIFT : PAGE_SHIFT;
+#endif /* CONFIG_MM_MODULES */
 	return 0;
 
 err:
diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index 535e763..4c11f2f 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -1915,6 +1915,33 @@ static int elf_core_dump(struct coredump_params *cprm)
 	if (gate_vma != NULL)
 		segs++;
 
+#ifdef CONFIG_MM_MODULES
+	/* Find out how many extra regions (if any) there are in vmas: */
+	for (vma = first_vma(current, gate_vma); vma != NULL;
+		vma = next_vma(vma, gate_vma)) {
+		if (vma->mm_module_ops) {
+			int nr_regions = 0;
+			unsigned long range_start = vma->vm_start;
+			unsigned long range_end;
+			do {
+				BUG_ON(!vma->mm_module_ops->probe_mapped);
+				if (vma->mm_module_ops->probe_mapped(vma,
+							range_start,
+							&range_end, NULL) &&
+				    vma_dump_size(vma, current->mm->flags) > 0)
+					nr_regions++;
+			} while (range_start = range_end, range_start <
+					vma->vm_end);
+			/*
+			 * Aadjust segment count according to # of regions in
+			 * vma. Note: this should decrement segment count for
+			 * vmas with no mapped regions.
+			 */
+			segs += nr_regions - 1;
+		}
+	}
+#endif /* CONFIG_MM_MODULES */
+
 	/* for notes section */
 	segs++;
 
@@ -1982,7 +2009,42 @@ static int elf_core_dump(struct coredump_params *cprm)
 	for (vma = first_vma(current, gate_vma); vma != NULL;
 			vma = next_vma(vma, gate_vma)) {
 		struct elf_phdr phdr;
+#if CONFIG_MM_MODULES
+		unsigned long range_start = vma->vm_start;
+		unsigned long range_end = vma->vm_end;
+		unsigned long range_vm_flags = vma->vm_flags;
+		do {
+			if (vma->mm_module_ops) {
+				BUG_ON(!vma->mm_module_ops->probe_mapped);
+				if (!vma->mm_module_ops->
+				    probe_mapped(vma, range_start, &range_end,
+						 &range_vm_flags) ||
+				    vma_dump_size(vma, cprm->mm_flags) == 0)
+					continue;
+			}
 
+			phdr.p_type = PT_LOAD;
+			phdr.p_offset = offset;
+			phdr.p_vaddr = range_start;
+			phdr.p_paddr = 0;
+			phdr.p_filesz = vma->mm_module_ops ?
+				(range_end - range_start) :
+				vma_dump_size(vma, cprm->mm_flags);
+			phdr.p_memsz = range_end - range_start;
+			offset += phdr.p_filesz;
+			phdr.p_flags = range_vm_flags & VM_READ ? PF_R : 0;
+			if (range_vm_flags & VM_WRITE)
+				phdr.p_flags |= PF_W;
+			if (range_vm_flags & VM_EXEC)
+				phdr.p_flags |= PF_X;
+			phdr.p_align = ELF_EXEC_PAGESIZE;
+
+			size += sizeof(phdr);
+			if (size > cprm->limit
+		    	|| !dump_write(cprm->file, &phdr, sizeof(phdr)))
+				goto end_coredump;
+		} while (range_start = range_end, range_start < vma->vm_end);
+#else /* CONFIG_MM_MODULES */
 		phdr.p_type = PT_LOAD;
 		phdr.p_offset = offset;
 		phdr.p_vaddr = vma->vm_start;
@@ -2001,6 +2063,7 @@ static int elf_core_dump(struct coredump_params *cprm)
 		if (size > cprm->limit
 		    || !dump_write(cprm->file, &phdr, sizeof(phdr)))
 			goto end_coredump;
+#endif /* CONFIG_MM_MODULES */
 	}
 
 	if (!elf_core_write_extra_phdrs(cprm->file, offset, &size, cprm->limit))
@@ -2022,6 +2085,43 @@ static int elf_core_dump(struct coredump_params *cprm)
 		unsigned long addr;
 		unsigned long end;
 
+#ifdef CONFIG_MM_MODULES
+		unsigned long range_start = vma->vm_start;
+		unsigned long range_end;
+		end = (vma->mm_module_ops ? vma->vm_end :
+		       vma->vm_start + vma_dump_size(vma, cprm->mm_flags));
+		range_end = end;
+
+		do {
+			if (vma->mm_module_ops) {
+				BUG_ON(!vma->mm_module_ops->probe_mapped);
+				if (!vma->mm_module_ops->
+				    probe_mapped(vma, range_start, &range_end,
+						 NULL) || 
+				    vma_dump_size(vma, cprm->mm_flags) == 0)
+					continue;
+			}
+
+			for (addr = range_start; addr < range_end;
+					addr += PAGE_SIZE) {
+				struct page *page;
+				int stop;
+
+				page = get_dump_page(addr);
+				if (page) {
+					void *kaddr = kmap(page);
+					stop = ((size += PAGE_SIZE) > cprm->limit) ||
+						!dump_write(cprm->file, kaddr,
+								PAGE_SIZE);
+					kunmap(page);
+					page_cache_release(page);
+				} else
+					stop = !dump_seek(cprm->file, PAGE_SIZE);
+				if (stop)
+					goto end_coredump;
+			}
+		} while (range_start = range_end, range_start < end);
+#else /* CONFIG_MM_MODULES */
 		end = vma->vm_start + vma_dump_size(vma, cprm->mm_flags);
 
 		for (addr = vma->vm_start; addr < end; addr += PAGE_SIZE) {
@@ -2041,6 +2141,7 @@ static int elf_core_dump(struct coredump_params *cprm)
 			if (stop)
 				goto end_coredump;
 		}
+#endif /* CONFIG_MM_MODULES */
 	}
 
 	if (!elf_core_write_extra_data(cprm->file, &size, cprm->limit))
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 47f5b14..3631572 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -220,6 +220,18 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
 		pgoff = ((loff_t)vma->vm_pgoff) << PAGE_SHIFT;
 	}
 
+#ifdef CONFIG_MM_MODULES
+	seq_printf(m, "%08lx-%08lx %c%c%c%c%c %08llx %02x:%02x %lu %n",
+			vma->vm_start,
+			vma->vm_end,
+			flags & VM_READ ? 'r' : '-',
+			flags & VM_WRITE ? 'w' : '-',
+			flags & VM_EXEC ? 'x' : '-',
+			flags & VM_MAYSHARE ? 's' : 'p',
+			vma->mm_module_ops ? 'm' : '\0',
+			((loff_t)vma->vm_pgoff) << PAGE_SHIFT,
+			MAJOR(dev), MINOR(dev), ino, &len);
+#else /* CONFIG_MM_MODULES */
 	seq_printf(m, "%08lx-%08lx %c%c%c%c %08llx %02x:%02x %lu %n",
 			vma->vm_start,
 			vma->vm_end,
@@ -229,6 +241,7 @@ static void show_map_vma(struct seq_file *m, struct vm_area_struct *vma)
 			flags & VM_MAYSHARE ? 's' : 'p',
 			pgoff,
 			MAJOR(dev), MINOR(dev), ino, &len);
+#endif /* CONFIG_MM_MODULES */
 
 	/*
 	 * Print the dentry name for named mappings, and a
@@ -389,6 +402,9 @@ static int show_smap(struct seq_file *m, void *v)
 	memset(&mss, 0, sizeof mss);
 	mss.vma = vma;
 	/* mmap_sem is held in m_start */
+#ifdef CONFIG_MM_MODULES
+	if (!vma->mm_module_ops)
+#endif /* CONFIG_MM_MODULES */
 	if (vma->vm_mm && !is_vm_hugetlb_page(vma))
 		walk_page_range(vma->vm_start, vma->vm_end, &smaps_walk);
 
@@ -503,6 +519,9 @@ static ssize_t clear_refs_write(struct file *file, const char __user *buf,
 		down_read(&mm->mmap_sem);
 		for (vma = mm->mmap; vma; vma = vma->vm_next) {
 			clear_refs_walk.private = vma;
+#ifdef CONFIG_MM_MODULES
+			if (!vma->mm_module_ops)
+#endif /* CONFIG_MM_MODULES */
 			if (is_vm_hugetlb_page(vma))
 				continue;
 			/*
@@ -618,6 +637,9 @@ static int pagemap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,
 		/* check that 'vma' actually covers this address,
 		 * and that it isn't a huge page vma */
 		if (vma && (vma->vm_start <= addr) &&
+#ifdef CONFIG_MM_MODULES
+		    !vma->mm_module_ops &&
+#endif /* CONFIG_MM_MODULES */
 		    !is_vm_hugetlb_page(vma)) {
 			pte = pte_offset_map(pmd, addr);
 			pfn = pte_to_pagemap_entry(*pte);
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 462acaf..6f3e1dc 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -226,6 +226,48 @@ struct vm_operations_struct {
 struct mmu_gather;
 struct inode;
 
+#ifdef CONFIG_PMEM_MODULES
+struct pmem_module_operations_struct {
+	int (*put_page)(struct page *page);
+	int (*get_page)(struct page *page);
+	int (*sparse_mem_map_populate)(unsigned long pnum, int nid);
+};
+#endif /* CONFIG_PMEM_MODULES */
+
+#ifdef CONFIG_MM_MODULES
+struct zap_details;
+
+struct mm_module_operations_struct {
+	int (*handle_mm_fault)(struct mm_struct *mm,
+			struct vm_area_struct *vma, unsigned long addr,
+			unsigned int flags);
+	int (*change_protection)(struct vm_area_struct *vma, unsigned long start,
+			unsigned long end, unsigned long newflags);
+	int (*copy_page_range)(struct mm_struct *dst_mm,
+			struct mm_struct *src_mm, struct vm_area_struct *vma);
+	int (*follow_page)(struct mm_struct *mm, struct vm_area_struct *vma,
+			struct page **pages, struct vm_area_struct **vmas,
+			unsigned long *position, int *length,
+			int i, int write);
+	int (*probe_mapped)(struct vm_area_struct *vma, unsigned long start,
+			unsigned long *end_range, unsigned long *range_vm_flags);
+	unsigned long (*unmap_page_range)(struct mmu_gather **tlbp,
+			struct vm_area_struct *vma, unsigned long addr,
+			unsigned long end, long *zap_work,
+			struct zap_details *details); 
+	void (*free_pgd_range)(struct mmu_gather *tlb, unsigned long addr,
+			unsigned long end, unsigned long floor,
+			unsigned long ceiling);
+	int (*init_module_vma)(struct vm_area_struct *vma,
+			struct vm_area_struct *old_vma);
+	void (*exit_module_vma)(struct vm_area_struct *vma);
+	int (*init_module_mm)(struct mm_struct *mm,
+			struct mm_module_struct *mm_mod);
+	int (*exit_module_mm)(struct mm_struct *mm,
+			struct mm_module_struct *mm_mod);
+};
+#endif /* CONFIG_MM_MODULES */
+
 #define page_private(page)		((page)->private)
 #define set_page_private(page, v)	((page)->private = (v))
 
@@ -235,6 +277,32 @@ struct inode;
  */
 #include <linux/page-flags.h>
 
+#ifdef CONFIG_PMEM_MODULES
+static inline void pmem_modules_get_page(struct page *page)
+{
+	struct pmem_module_struct *module = pmem_modules;
+	for (module = pmem_modules; module; module = module->next) {
+		VM_BUG_ON(!module->pmem_module_ops->get_page);
+		if (module->pmem_module_ops->get_page(page))
+			return;
+	}
+	/* One of the modules should have picked the page up */
+	BUG();
+}
+
+static inline void pmem_modules_put_page(struct page *page)
+{
+	struct pmem_module_struct *module = pmem_modules;
+	for (module = pmem_modules; module; module = module->next) {
+		VM_BUG_ON(!module->pmem_module_ops->put_page);
+		if (module->pmem_module_ops->put_page(page))
+			return;
+	}
+	/* One of the modules should have picked the page up */
+	BUG();
+}
+#endif /* CONFIG_PMEM_MODULES */
+
 /*
  * Methods to modify the page usage count.
  *
@@ -311,6 +379,12 @@ static inline int page_count(struct page *page)
 
 static inline void get_page(struct page *page)
 {
+#ifdef CONFIG_PMEM_MODULES
+	if (unlikely(PagePmemModule(page))) {
+		pmem_modules_get_page(page);
+		return;
+	}
+#endif /* CONFIG_PMEM_MODULES */
 	page = compound_head(page);
 	VM_BUG_ON(atomic_read(&page->_count) == 0);
 	atomic_inc(&page->_count);
@@ -715,7 +789,15 @@ static inline int page_mapped(struct page *page)
 #define VM_FAULT_NOPAGE	0x0100	/* ->fault installed the pte, not return page */
 #define VM_FAULT_LOCKED	0x0200	/* ->fault locked the returned page */
 
+#ifdef CONFIG_MM_MODULES
+#define VM_FAULT_SEGV_ACCERR	0x1000
+#define VM_FAULT_SEGV_MAPERR	0x2000
+#define VM_FAULT_SIGSEGV	(VM_FAULT_SEGV_ACCERR | VM_FAULT_SEGV_MAPERR)
+#define VM_FAULT_ERROR \
+	(VM_FAULT_OOM | VM_FAULT_SIGBUS | VM_FAULT_HWPOISON | VM_FAULT_SIGSEGV)
+#else /* !CONFIG_MM_MODULES */
 #define VM_FAULT_ERROR	(VM_FAULT_OOM | VM_FAULT_SIGBUS | VM_FAULT_HWPOISON)
+#endif /* CONFIG_MM_MODULES */
 
 /*
  * Can be called by the pagefault handler when it gets a VM_FAULT_OOM.
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index b8bb9a6..0dc8c1a 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -183,6 +183,10 @@ struct vm_area_struct {
 #ifdef CONFIG_NUMA
 	struct mempolicy *vm_policy;	/* NUMA policy for the VMA */
 #endif
+#ifdef CONFIG_MM_MODULES
+	struct mm_module_operations_struct * mm_module_ops;
+	void * mm_module_vma_state;
+#endif /* CONFIG_MM_MODULES */
 };
 
 struct core_thread {
@@ -219,6 +223,23 @@ struct mm_rss_stat {
 };
 #endif /* !USE_SPLIT_PTLOCKS */
 
+#ifdef CONFIG_PMEM_MODULES
+struct pmem_module_struct {
+	struct pmem_module_operations_struct * pmem_module_ops;
+	struct pmem_module_struct *next;
+};
+
+extern struct pmem_module_struct *pmem_modules;
+#endif /* CONFIG_PMEM_MODULES */
+
+#ifdef CONFIG_MM_MODULES
+struct mm_module_struct {
+	struct mm_module_operations_struct * mm_module_ops;
+	void * mm_module_mm_state;
+	struct mm_module_struct *next;
+};
+#endif /* CONFIG_MM_MODULES */
+
 struct mm_struct {
 	struct vm_area_struct * mmap;		/* list of VMAs */
 	struct rb_root mm_rb;
@@ -310,6 +331,9 @@ struct mm_struct {
 #ifdef CONFIG_MMU_NOTIFIER
 	struct mmu_notifier_mm *mmu_notifier_mm;
 #endif
+#ifdef CONFIG_MM_MODULES
+	struct mm_module_struct *mm_modules;
+#endif /* CONFIG_MM_MODULES */
 };
 
 /* Future-safe accessor for struct mm_struct's cpu_vm_mask. */
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 5b59f35..d8d5db2 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -108,6 +108,9 @@ enum pageflags {
 #ifdef CONFIG_MEMORY_FAILURE
 	PG_hwpoison,		/* hardware poisoned page. Don't touch */
 #endif
+#ifdef CONFIG_PMEM_MODULES
+	PG_pmem_module,
+#endif /* CONFIG_PMEM_MODULES */
 	__NR_PAGEFLAGS,
 
 	/* Filesystems */
@@ -284,6 +287,12 @@ PAGEFLAG_FALSE(HWPoison)
 
 u64 stable_page_flags(struct page *page);
 
+#ifdef CONFIG_PMEM_MODULES
+PAGEFLAG(PmemModule, pmem_module)
+#else /* !CONFIG_PMEM_MODULES */
+PAGEFLAG_FALSE(PmemModule)
+#endif /* CONFIG_PMEM_MODULES */
+
 static inline int PageUptodate(struct page *page)
 {
 	int ret = test_bit(PG_uptodate, &(page)->flags);
diff --git a/kernel/fork.c b/kernel/fork.c
index 4c14942..08c2f33 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -366,6 +366,13 @@ static int dup_mmap(struct mm_struct *mm, struct mm_struct *oldmm)
 			spin_unlock(&mapping->i_mmap_lock);
 		}
 
+#ifdef CONFIG_MM_MODULES
+		if (mpnt->mm_module_ops) {
+			BUG_ON(!mpnt->mm_module_ops->init_module_vma);
+			mpnt->mm_module_ops->init_module_vma(tmp, mpnt);
+		}
+#endif /* CONFIG_MM_MODULES */
+
 		/*
 		 * Clear hugetlb-related page reserves for children. This only
 		 * affects MAP_PRIVATE mappings. Faults generated by the child
@@ -456,6 +463,41 @@ static void mm_init_aio(struct mm_struct *mm)
 #endif
 }
 
+#ifdef CONFIG_MM_MODULES
+void mm_modules_init(struct mm_struct * mm)
+{
+	struct mm_module_struct *old_mm_modules;
+	struct mm_module_struct *module;
+
+	/* Extract old mm's modules list, initialize new mm's list: */
+	old_mm_modules = mm->mm_modules;
+	mm->mm_modules = NULL;
+
+	/* Iterate on modules. Allow each to initialize new mm based on old: */
+	for (module = old_mm_modules; module; module = module->next) {
+		BUG_ON(!module->mm_module_ops);
+		BUG_ON(!module->mm_module_ops->init_module_mm);
+		module->mm_module_ops->init_module_mm(mm, module);
+	}
+}
+
+void mm_modules_exit(struct mm_struct * mm)
+{
+	struct mm_module_struct *module;
+
+	/*
+	 * Modules will remove their own mm_module_struct and free it,
+	 * so keep calling the top module's mm_exit_module call
+	 * until none are left.
+	 */
+	while ((module = mm->mm_modules)) {
+		BUG_ON(!module->mm_module_ops);
+		BUG_ON(!module->mm_module_ops->exit_module_mm);
+		module->mm_module_ops->exit_module_mm(mm, module);
+	}
+}
+#endif /* CONFIG_MM_MODULES */
+
 static struct mm_struct * mm_init(struct mm_struct * mm, struct task_struct *p)
 {
 	atomic_set(&mm->mm_users, 1);
@@ -473,6 +515,10 @@ static struct mm_struct * mm_init(struct mm_struct * mm, struct task_struct *p)
 	mm_init_aio(mm);
 	mm_init_owner(mm, p);
 
+#ifdef CONFIG_MM_MODULES
+	mm_modules_init(mm);
+#endif /* CONFIG_MM_MODULES */
+
 	if (likely(!mm_alloc_pgd(mm))) {
 		mm->def_flags = 0;
 		mmu_notifier_mm_init(mm);
@@ -524,6 +570,9 @@ void mmput(struct mm_struct *mm)
 		exit_aio(mm);
 		ksm_exit(mm);
 		exit_mmap(mm);
+#ifdef CONFIG_MM_MODULES
+		mm_modules_exit(mm);
+#endif /* CONFIG_MM_MODULES */
 		set_mm_exe_file(mm, NULL);
 		if (!list_empty(&mm->mmlist)) {
 			spin_lock(&mmlist_lock);
diff --git a/mm/Kconfig b/mm/Kconfig
index 9c61158..f4afb58 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -128,6 +128,20 @@ config SPARSEMEM_VMEMMAP
 	 pfn_to_page and page_to_pfn operations.  This is the most
 	 efficient option when sufficient kernel resources are available.
 
+config MM_MODULES
+	bool "Memory Management Modules"
+	depends on MMU_NOTIFIER
+	default y
+	help
+	 Provides support for dynamically loadable memory management modules
+
+config PMEM_MODULES
+	bool "Physical Memory Management Modules"
+	default y
+	help
+	 Provides support for dynamically loadable physical memory
+	 management modules
+
 # eventually, we can have this option just 'select SPARSEMEM'
 config MEMORY_HOTPLUG
 	bool "Allow for memory hot-add"
diff --git a/mm/Makefile b/mm/Makefile
index 6c2a73a..00b302f 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -44,3 +44,4 @@ obj-$(CONFIG_MEMORY_FAILURE) += memory-failure.o
 obj-$(CONFIG_HWPOISON_INJECT) += hwpoison-inject.o
 obj-$(CONFIG_DEBUG_KMEMLEAK) += kmemleak.o
 obj-$(CONFIG_DEBUG_KMEMLEAK_TEST) += kmemleak-test.o
+obj-$(CONFIG_MM_MODULES) += mm_module_exports.o
diff --git a/mm/memory.c b/mm/memory.c
index 833952d..0130cac 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -67,6 +67,11 @@
 
 #include "internal.h"
 
+#ifdef CONFIG_PMEM_MODULES
+struct pmem_module_struct *pmem_modules = NULL;
+EXPORT_SYMBOL_GPL(pmem_modules);
+#endif /* CONFIG_PMEM_MODULES */
+
 #ifndef CONFIG_NEED_MULTIPLE_NODES
 /* use the per-pgdat data instead for discontigmem - mbligh */
 unsigned long max_mapnr;
@@ -375,6 +380,14 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma,
 		unlink_anon_vmas(vma);
 		unlink_file_vma(vma);
 
+#ifdef CONFIG_MM_MODULES
+		if (vma->mm_module_ops) {
+			BUG_ON(!vma->mm_module_ops->free_pgd_range);
+			vma->mm_module_ops->free_pgd_range(tlb, addr,
+					vma->vm_end, floor,
+					next? next->vm_start: ceiling);
+		} else 
+#endif /* CONFIG_MM_MODULES */
 		if (is_vm_hugetlb_page(vma)) {
 			hugetlb_free_pgd_range(tlb, addr, vma->vm_end,
 				floor, next? next->vm_start: ceiling);
@@ -383,6 +396,9 @@ void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *vma,
 			 * Optimization: gather nearby vmas into one call down
 			 */
 			while (next && next->vm_start <= vma->vm_end + PMD_SIZE
+#ifdef CONFIG_MM_MODULES
+				&& !next->mm_module_ops
+#endif /* CONFIG_MM_MODULES */
 			       && !is_vm_hugetlb_page(next)) {
 				vma = next;
 				next = vma->vm_next;
@@ -843,6 +859,13 @@ int copy_page_range(struct mm_struct *dst_mm, struct mm_struct *src_mm,
 	 * readonly mappings. The tradeoff is that copy_page_range is more
 	 * efficient than faulting.
 	 */
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops) {
+		BUG_ON(!vma->mm_module_ops->copy_page_range);
+		return vma->mm_module_ops->copy_page_range(dst_mm, src_mm, vma);
+	}
+#endif /* CONFIG_MM_MODULES */
+
 	if (!(vma->vm_flags & (VM_HUGETLB|VM_NONLINEAR|VM_PFNMAP|VM_INSERTPAGE))) {
 		if (!vma->anon_vma)
 			return 0;
@@ -1131,6 +1154,14 @@ unsigned long unmap_vmas(struct mmu_gather **tlbp,
 				tlb_start_valid = 1;
 			}
 
+#ifdef CONFIG_MM_MODULES
+			if (unlikely(vma->mm_module_ops)) {
+				BUG_ON(!vma->mm_module_ops->unmap_page_range);
+				start = vma->mm_module_ops->unmap_page_range(
+							tlbp, vma, start, end,
+							&zap_work, details);
+			} else
+#endif /* CONFIG_MM_MODULES */
 			if (unlikely(is_vm_hugetlb_page(vma))) {
 				/*
 				 * It is undesirable to test vma->vm_file as it
@@ -1404,6 +1435,15 @@ int __get_user_pages(struct task_struct *tsk, struct mm_struct *mm,
 		    !(vm_flags & vma->vm_flags))
 			return i ? : -EFAULT;
 
+#ifdef CONFIG_MM_MODULES
+		if (vma->mm_module_ops) {
+			BUG_ON(!vma->mm_module_ops->follow_page);
+			i = vma->mm_module_ops->follow_page(mm, vma, pages,
+					vmas, &start, &nr_pages, i, gup_flags);
+			continue;
+		}
+#endif /* CONFIG_MM_MODULES */
+
 		if (is_vm_hugetlb_page(vma)) {
 			i = follow_hugetlb_page(mm, vma, pages, vmas,
 					&start, &nr_pages, i, gup_flags);
@@ -3122,6 +3162,14 @@ int handle_mm_fault(struct mm_struct *mm, struct vm_area_struct *vma,
 	/* do counter updates before entering really critical section. */
 	check_sync_rss_stat(current);
 
+#ifdef CONFIG_MM_MODULES
+	if (unlikely(vma->mm_module_ops)) {
+		BUG_ON(!vma->mm_module_ops->handle_mm_fault);
+		return vma->mm_module_ops->handle_mm_fault(mm, vma,
+				address, flags);
+	}
+#endif /* CONFIG_MM_MODULES */
+
 	if (unlikely(is_vm_hugetlb_page(vma)))
 		return hugetlb_fault(mm, vma, address, flags);
 
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 08f40a2..d359958 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -519,6 +519,9 @@ check_range(struct mm_struct *mm, unsigned long start, unsigned long end,
 				return ERR_PTR(-EFAULT);
 		}
 		if (!is_vm_hugetlb_page(vma) &&
+#ifdef CONFIG_MM_MODULES
+		    !vma->mm_module_ops &&
+#endif /* CONFIG_MM_MODULES */
 		    ((flags & MPOL_MF_STRICT) ||
 		     ((flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)) &&
 				vma_migratable(vma)))) {
@@ -2480,6 +2483,11 @@ int show_numa_map(struct seq_file *m, void *v)
 		seq_printf(m, " stack");
 	}
 
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops) {
+		seq_printf(m, " mm_module");
+	} else
+#endif /* CONFIG_MM_MODULES */
 	if (is_vm_hugetlb_page(vma)) {
 		check_huge_range(vma, vma->vm_start, vma->vm_end, md);
 		seq_printf(m, " huge");
@@ -2507,6 +2515,9 @@ int show_numa_map(struct seq_file *m, void *v)
 		seq_printf(m," swapcache=%lu", md->swapcache);
 
 	if (md->active < md->pages && !is_vm_hugetlb_page(vma))
+#ifdef CONFIG_MM_MODULES
+		if (!vma->mm_module_ops)
+#endif /* CONFIG_MM_MODULES */
 		seq_printf(m," active=%lu", md->active);
 
 	if (md->writeback)
diff --git a/mm/mlock.c b/mm/mlock.c
index 8f4e2df..c8de2bc 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -264,6 +264,9 @@ long mlock_vma_pages_range(struct vm_area_struct *vma,
 		goto no_mlock;
 
 	if (!((vma->vm_flags & (VM_DONTEXPAND | VM_RESERVED)) ||
+#ifdef CONFIG_MM_MODULES
+			(vma->mm_module_ops) ||
+#endif /* CONFIG_MM_MODULES */
 			is_vm_hugetlb_page(vma) ||
 			vma == get_gate_vma(current))) {
 
@@ -364,6 +367,9 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev,
 		goto out;	/* don't set VM_LOCKED,  don't count */
 
 	if ((vma->vm_flags & (VM_DONTEXPAND | VM_RESERVED)) ||
+#ifdef CONFIG_MM_MODULES
+			(vma->mm_module_ops) ||
+#endif /* CONFIG_MM_MODULES */
 			is_vm_hugetlb_page(vma) ||
 			vma == get_gate_vma(current)) {
 		if (lock)
diff --git a/mm/mm_module_exports.c b/mm/mm_module_exports.c
new file mode 100644
index 0000000..b08d41e
--- /dev/null
+++ b/mm/mm_module_exports.c
@@ -0,0 +1,100 @@
+
+#include <linux/module.h>
+
+#include <linux/errno.h>
+#include <linux/mm.h>
+#include <linux/elf.h>
+#include <asm/pgtable.h>
+
+/* arch/x86/kernel/tlb_64.c */
+extern void flush_tlb_page(struct vm_area_struct *vma, unsigned long va);
+EXPORT_SYMBOL_GPL(flush_tlb_page);
+extern void flush_tlb_mm(struct mm_struct *);
+EXPORT_SYMBOL_GPL(flush_tlb_mm);
+
+/* arch/x86/mm/pgtable.c */
+extern void ___pud_free_tlb(struct mmu_gather *, pud_t *);
+extern void ___pmd_free_tlb(struct mmu_gather *, pmd_t *);
+EXPORT_SYMBOL_GPL(___pud_free_tlb);
+EXPORT_SYMBOL_GPL(___pmd_free_tlb);
+
+/* XXX Not sure how the heck _this_ is supposed to get set when building a KO... */
+#ifndef __PAGETABLE_PUD_FOLDED
+extern int __pud_alloc(struct mm_struct *, pgd_t *, unsigned long);
+EXPORT_SYMBOL_GPL(__pud_alloc);
+#endif
+
+/* mm/swap.c */
+extern void lru_add_drain(void);
+EXPORT_SYMBOL_GPL(lru_add_drain);
+
+/* mm/memory.c */
+extern void pgd_clear_bad(pgd_t *);
+EXPORT_SYMBOL_GPL(pgd_clear_bad);
+extern void pud_clear_bad(pud_t *);
+EXPORT_SYMBOL_GPL(pud_clear_bad);
+struct page *vm_normal_page(struct vm_area_struct *, unsigned long, pte_t pte);
+EXPORT_SYMBOL_GPL(vm_normal_page);
+extern int __pmd_alloc(struct mm_struct *, pud_t *, unsigned long);
+EXPORT_SYMBOL_GPL(__pmd_alloc);
+extern void pmd_clear_bad(pmd_t *);
+EXPORT_SYMBOL_GPL(pmd_clear_bad);
+#if defined(SPLIT_RSS_COUNTING)
+extern unsigned long get_mm_counter(struct mm_struct *mm, int member);
+EXPORT_SYMBOL_GPL(get_mm_counter);
+#endif /* defined(SPLIT_RSS_COUNTING) */
+
+/* mm/swap_state.c */
+extern void free_pages_and_swap_cache(struct page **, int);
+EXPORT_SYMBOL_GPL(free_pages_and_swap_cache);
+
+#ifdef CONFIG_MMU_NOTIFIER
+/* mm/mmu_notifier.c */
+extern void __mmu_notifier_invalidate_range_start(struct mm_struct *,
+		unsigned long, unsigned long);
+EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range_start);
+extern void __mmu_notifier_invalidate_range_end(struct mm_struct *,
+		unsigned long, unsigned long);
+EXPORT_SYMBOL_GPL(__mmu_notifier_invalidate_range_end);
+#endif /* CONFIG_MMU_NOTIFIER */
+
+/* mm/rmap.c */
+extern int anon_vma_prepare(struct vm_area_struct *);
+EXPORT_SYMBOL_GPL(anon_vma_prepare);
+
+/* mm/bootmem.c */
+extern unsigned long max_pfn;
+EXPORT_SYMBOL_GPL(max_pfn);
+
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
+/* arch/x86/mm/init_64.c */
+extern int vmemmap_populate(struct page *, unsigned long, int);
+EXPORT_SYMBOL_GPL(vmemmap_populate);
+#endif
+
+#if 1
+/* WTF? */
+/* arch/x86/mm/init.c */
+extern struct mmu_gather mmu_gathers;
+EXPORT_SYMBOL_GPL(mmu_gathers);
+#endif
+
+/* kernel/fork.c */
+extern void __put_task_struct(struct task_struct *);
+EXPORT_SYMBOL_GPL(__put_task_struct);
+extern rwlock_t tasklist_lock;
+EXPORT_SYMBOL_GPL(tasklist_lock);
+
+/* kernel/pid.c */
+enum pid_type;
+extern struct task_struct *get_pid_task(struct pid *, enum pid_type);
+EXPORT_SYMBOL_GPL(get_pid_task);
+extern struct task_struct *find_task_by_vpid(pid_t vnr);
+EXPORT_SYMBOL_GPL(find_task_by_vpid);
+
+/* arch/x86/kernel/init_task.c */
+extern struct mm_struct init_mm;
+EXPORT_SYMBOL_GPL(init_mm); /* will be removed in 2.6.26 */
+
+/* mm/mmap.c */
+EXPORT_SYMBOL_GPL(split_vma);
diff --git a/mm/mmap.c b/mm/mmap.c
index 456ec6f..0c41247 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -237,6 +237,12 @@ static struct vm_area_struct *remove_vma(struct vm_area_struct *vma)
 		if (vma->vm_flags & VM_EXECUTABLE)
 			removed_exe_file_vma(vma->vm_mm);
 	}
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops) {
+		BUG_ON(!vma->mm_module_ops->exit_module_vma);
+		vma->mm_module_ops->exit_module_vma(vma);
+	}
+#endif /* CONFIG_MM_MODULES */
 	mpol_put(vma_policy(vma));
 	kmem_cache_free(vm_area_cachep, vma);
 	return next;
@@ -661,6 +667,10 @@ static inline int is_mergeable_vma(struct vm_area_struct *vma,
 		return 0;
 	if (vma->vm_ops && vma->vm_ops->close)
 		return 0;
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops)
+		return 0;
+#endif /* CONFIG_MM_MODULES */
 	return 1;
 }
 
@@ -1927,6 +1937,11 @@ static int __split_vma(struct mm_struct * mm, struct vm_area_struct * vma,
 	struct vm_area_struct *new;
 	int err = -ENOMEM;
 
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops)
+		return -EINVAL;
+#endif /* CONFIG_MM_MODULES */
+
 	if (is_vm_hugetlb_page(vma) && (addr &
 					~(huge_page_mask(hstate_vma(vma)))))
 		return -EINVAL;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 2d1bf7c..a3bd604 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -189,6 +189,21 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
 			goto fail;
 	}
 
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops) {
+		BUG_ON(!vma->mm_module_ops->change_protection);
+		error = vma->mm_module_ops->change_protection(vma, start,
+				end, newflags);
+		if (error)
+			goto fail;
+		/* mm_module_ops->change_protection is responsible for
+		 * vma flags, vm_page_prot, all pte settings, mmu_notifier, 
+		 * and vm_stat accounting. When it comes back, we're done.
+		 */
+		return 0;
+	} 
+#endif /* CONFIG_MM_MODULES */
+
 success:
 	/*
 	 * vm_flags and vm_page_prot are protected by the mmap_sem
diff --git a/mm/mremap.c b/mm/mremap.c
index cde56ee..ab69286 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -269,6 +269,11 @@ static struct vm_area_struct *vma_to_resize(unsigned long addr,
 	if (!vma || vma->vm_start > addr)
 		goto Efault;
 
+#ifdef CONFIG_MM_MODULES
+	if (vma->mm_module_ops)
+		goto Einval;
+#endif /* CONFIG_MM_MODULES */
+
 	if (is_vm_hugetlb_page(vma))
 		goto Einval;
 
diff --git a/mm/sparse.c b/mm/sparse.c
index dc0cc4d..c8afd16 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -612,11 +612,32 @@ void __init sparse_init(void)
 	free_bootmem(__pa(usemap_map), size);
 }
 
+#ifdef CONFIG_PMEM_MODULES
+static inline int pmem_modules_sparse_mem_map_populate(unsigned long pnum,
+		int nid)
+{
+	struct pmem_module_struct *pmem_module = pmem_modules;
+	int success = true;
+	while (pmem_module) {
+		BUG_ON(!pmem_module->pmem_module_ops->sparse_mem_map_populate);
+		success &=
+			pmem_module->pmem_module_ops->sparse_mem_map_populate(
+					pnum, nid);
+		pmem_module = pmem_module->next;
+	}
+	return success;
+}
+#endif  /* CONFIG_PMEM_MODULES */
+
 #ifdef CONFIG_MEMORY_HOTPLUG
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
 static inline struct page *kmalloc_section_memmap(unsigned long pnum, int nid,
 						 unsigned long nr_pages)
 {
+#ifdef CONFIG_PMEM_MODULES
+	if (!pmem_modules_sparse_mem_map_populate(pnum, nid))
+		return NULL;
+#endif /* CONFIG_PMEM_MODULES */
 	/* This will make the necessary allocations eventually. */
 	return sparse_mem_map_populate(pnum, nid);
 }
diff --git a/mm/swap.c b/mm/swap.c
index 7cd60bf..e903a54 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -72,6 +72,12 @@ static void put_compound_page(struct page *page)
 
 void put_page(struct page *page)
 {
+#ifdef CONFIG_PMEM_MODULES
+	if (unlikely(PagePmemModule(page))) {
+		pmem_modules_put_page(page);
+		return;
+	}
+#endif /* CONFIG_PMEM_MODULES */
 	if (unlikely(PageCompound(page)))
 		put_compound_page(page);
 	else if (put_page_testzero(page))
@@ -335,6 +341,16 @@ void release_pages(struct page **pages, int nr, int cold)
 	pagevec_init(&pages_to_free, cold);
 	for (i = 0; i < nr; i++) {
 		struct page *page = pages[i];
+#ifdef CONFIG_PMEM_MODULES
+		if (unlikely(PagePmemModule(page))) {
+			if (zone) {
+				spin_unlock_irqrestore(&zone->lru_lock, flags);
+				zone = NULL;
+			}
+			pmem_modules_put_page(page);
+			continue;
+		}
+#endif /* CONFIG_PMEM_MODULES */
 
 		if (unlikely(PageCompound(page))) {
 			if (zone) {